645.769.83 - System Test & Evaluation

Course Structure

The course materials are divided into modules which can be accessed by clicking Course Modules on the left menu of Blackboard. A module will have several sections including the overview, content, readings, discussions, and assignments. You are encouraged to preview all sections of the module before starting. Most modules run for a period of seven (7) days, exceptions are noted on the Course Outline. Please regularly check the Course Outline and Announcements for assignment due dates.

Course Topics

Course Goals

By the end of this course, you will be able to create a test and evaluation approach that results in fielding an effective, interoperable system that meets requirements.

Course Learning Outcomes (CLOs)

• 1. Describe a system test and evaluation process that reduces risk and is focused on assessing how the system meets requirements and how well it performs.
• 2.. Define test objectives, test concepts, and the method of verification for each system requirement to assure test objectives and concepts are traceable to requirements.
• 3. Create an integration and test approach and test cases to describe how the system will be built and tested.
• 4. Create a qualification test approach and test cases (cross referenced to system requirements) to inform a system acceptance decision.
• 5. Develop system test and evaluation scenarios, field tests, and acceptance criteria for operational states and environmental factors.

Textbooks

No textbook required.  

Other Materials & Online Resources

Online references and resources are available.

Required Software

Please click on the link to the SE Toolkit in the Blackboard menu to see several options for software tools available for Model Based Systems Engineering (MBSE). 

In Module 1, under E-Reserves, you will find a link to tutorials for using Vitech CORE.  This tool will be discussed during this course as an option for the group project.  Students are encouraged to download and learn this tool to use in their capstone project.

Student Coursework Requirements

It is expected that each module will take approximately 7–10 hours per week to complete. Here is an approximate breakdown: reading the assigned sections of the texts (approximately 3–4 hours per week) as well as some outside reading, listening to the audio annotated slide presentations (approximately 2–3 hours per week), and writing assignments (approximately 2–3 hours per week).

It is expected that each module will take approximately at least 10 hours per week to complete if students have a strong foundation in the fundamentals from previous courses or on the job experience.  More time may be required to refresh prior learning necessary to digest course content and complete assignments.  Recorded lectures, E-Reserves readings, and written assignments are listed for each module.   Most of the reading is supplemental and intended as a resource for students.  If required, it will be called out in the assignment.

This course will consist of four basic student requirements, described in detail on the next pages:

1 - Preparation and Participation (Quizzes/Discussions, 18% of Final Grade Calculation)

Please carefully review lectures and reading material to be prepared for quizzes and discussions. The majority of readings are supplemental, a resource to use if needed.  Required readings are called out in the assignments.

QUIZZES

The purpose of weekly quizzes is to help students keep on track with digesting and learning course content.  Please submit your weekly Quiz by the evening of day 3 for that module week.  Grades are automatically calculated based on points for each answer.  A 5-point late penalty is assessed for each week late.

DISCUSSIONS

The purpose of discussions is to interact with other students about a specific topic.  Please include meaningful insights in your postings and responses to your classmates.  Feel free to agree or disagree with your classmates in a way that is civil and constructive.  The instructor(s) for your section will monitor discussions and provide feedback online in the discussion forums. 

There are two parts you must complete for each discussion, each worth 50% of your grade:

• PART ONE: Post your initial response to the discussion questions by the evening of day 4 for that module week. Posting a response to the discussion question is part one of your grade for module.  Not all questions require references to support your answer, but where sources are drawn upon please cite the source.  The initial posting is 50% of your grade.  A 5-point late penalty is assessed for each week late.

• PART TWO: Part two of your grade for module discussion is your interaction (i.e., responding to classmate postings with thoughtful responses) with at least two classmates.  Reply to at least TWO of the postings of your colleagues by the evening of day 7 of that module week.  Posting at least TWO responses is 50% of your grade.  A 5-point late penalty is assessed for each week late.  

For discussions, the following grading criteria are assessed by your instructor for Parts One and Two:

• 100–90 = A—Timeliness [regularly participates; all required postings; early in discussion; throughout the discussion]; Critical Thinking [rich in content; full of thoughts, insight, and analysis].
• 89–80 = B—Timeliness [frequently participates; all required postings; some not in time for others to read and respond]; Critical Thinking [substantial information; thought, insight, and analysis has taken place].
• 79–70 = C—Timeliness [infrequently participates; all required postings; most at the last minute without allowing for response time]; Critical Thinking [generally competent; information is thin and commonplace].
• < 70 = D or < 63 = F—Timeliness [rarely participates; some, or all required postings missing]; Critical Thinking [rudimentary and superficial; no analysis or insight is displayed].

2 - Homework Assignments (32% of Final Grade Calculation)

Assignments may include qualitative questions or essays (e.g. narratives, literature reviews, model summaries), or quantitative problem sets and case study assessments.  Please include your name and an assignment identifier on each submission, along with your name and a page number on each page.

Due dates are listed in the Course Outline.  Late submissions will be reduced by one letter grade for each week late (no exceptions without prior coordination with the instructors). 

If, after submitting a written assignment you receive a grade less than an 85, you are encouraged to redo the assignment and resubmit it. If the resubmission results in a better grade, that grade will be substituted for the previous grade minus 5 points for the resubmit.

A grading rubric is included with each homework that shows how each question or item is weighted, and what is expected to address that particular question or item. 

3 - Course Exam (15% of Final Grade Calculation)

The exam will include qualitative questions (e.g. narratives about lessons learned and their significance), a few quantitative problem sets (calculations or essays on quantitative methods), and multiple choice questions.  Please include your name and an assignment identifier on each submission, along with your name and a page number on each page.

The exam will be due at the end of Module 8.  Due dates are listed in the Course Outline.   Late assignments or resubmissions are not accepted for the exam. 

A grading rubric is included with the exam that shows how each question or item is weighted, and what is expected to address that particular question or item. 

4 - Group Project (30% of Final Grade Calculation) and Team/Self-Assessment (5% of Final Grade Calculation)

A Group Project will be assigned several weeks into the course which will continue until the end of the course.  

The Group Project Final Report and Team /Self-Assessment are due the week before the end of the course (Module 13). Late assignments are not accepted. 

Team presentations are scheduled in the last week of the course (Module 14).  Presentations are not graded and provide teams with an opportunity to share their approach with other teams.  Online sessions share their test approaches via an Office Hours Zoom session on the last evening of the course. 

A detailed grading rubric is provided with the project description and artifacts.  The rubric weighs products generated by the team and Is used to specify where points are earned for each product in the report. The following grading criteria are considered for each item in the rubric as part of the evaluation:

1. Student preparation and participation as a contributor to products generated as part of a system test approach described in more detail in the Group Project description. (40%)
2. Student technical understanding of the test approach and their individual role as a contributor. (20%)
3. Team preparation and participation in peer reviews of products, drafting interim products for instructor review, and preparing the final report for submission. (20%)
4. Team technical understanding of key activities in the test approach as related to the team role assumed by the team as supplier of the test approach and the instructor role as the customer or acquirer. (20%)

For the Team/Self-Assessment, each student assesses each team member’s performance and self-assesses their impact on the team’s performance.  Late assignments are not accepted. The grade is based on four elements as follows:

• Team member ratings of student performance (25%)
• Instructor rating of student performance (25%)
• Student’s assessment of each team member (25%)
• Student’s own self-assessment (25% of grade).

Ratings are on a scale of 1 – 5, with 5 being the highest rating. Five criteria are used: 1 - availability, 2- overall contributions to the project’s technical product, 3 - timeliness, 4 - quality, and 5 - quantity.  

Thoughtful insightful comments shall be provided to explain your ratings and to complete your assessment of team members and yourself. Points are deducted if comments are missing or do not fully explain your ratings.

Please refer to the lecture in Module 13 for specific details on how to conduct your team member assessment and self- assessment.

Grading Policy

Assignments are due according to the dates posted in your Blackboard course site on the Course Outline. Instructors post grades one week after assignment due dates.

Instructors generally do not directly grade spelling and grammar. However, egregious violations of the rules of the English language will be noted without comment. Consistently poor performance in either spelling or grammar is taken as an indication of poor written communication ability that may detract from your grade.

A grade of A indicates achievement of consistent excellence and distinction throughout the course—that is, conspicuous excellence in all aspects of assignments and discussion in every week.

A grade of B indicates work that meets all course requirements on a level appropriate for graduate academic work. These criteria apply to both undergraduates and graduate students taking the course.

EP uses a +/- grading system (see “Grading System”, Graduate Programs catalog, p. 10).

Course Evaluation

The following weighting will determine final grades:

Academic Policies

Deadlines for Adding, Dropping and Withdrawing from Courses

Students may add a course up to one week after the start of the term for that particular course. Students may drop courses according to the drop deadlines outlined in the EP academic calendar (https://ep.jhu.edu/student-services/academic-calendar/). Between the 6th week of the class and prior to the final withdrawal deadline, a student may withdraw from a course with a W on their academic record. A record of the course will remain on the academic record with a W appearing in the grade column to indicate that the student registered and withdrew from the course.

Academic Misconduct Policy

All students are required to read, know, and comply with the Johns Hopkins University Krieger School of Arts and Sciences (KSAS) / Whiting School of Engineering (WSE) Procedures for Handling Allegations of Misconduct by Full-Time and Part-Time Graduate Students.

This policy prohibits academic misconduct, including but not limited to the following: cheating or facilitating cheating; plagiarism; reuse of assignments; unauthorized collaboration; alteration of graded assignments; and unfair competition. Course materials (old assignments, texts, or examinations, etc.) should not be shared unless authorized by the course instructor. Any questions related to this policy should be directed to EP’s academic integrity officer at ep-academic-integrity@jhu.edu.

Students with Disabilities - Accommodations and Accessibility

Johns Hopkins University values diversity and inclusion. We are committed to providing welcoming, equitable, and accessible educational experiences for all students. Students with disabilities (including those with psychological conditions, medical conditions and temporary disabilities) can request accommodations for this course by providing an Accommodation Letter issued by Student Disability Services (SDS). Please request accommodations for this course as early as possible to provide time for effective communication and arrangements.

For further information or to start the process of requesting accommodations, please contact Student Disability Services at Engineering for Professionals, ep-disability-svcs@jhu.edu.

Student Conduct Code

The fundamental purpose of the JHU regulation of student conduct is to promote and to protect the health, safety, welfare, property, and rights of all members of the University community as well as to promote the orderly operation of the University and to safeguard its property and facilities. As members of the University community, students accept certain responsibilities which support the educational mission and create an environment in which all students are afforded the same opportunity to succeed academically. 

For a full description of the code please visit the following website: https://studentaffairs.jhu.edu/policies-guidelines/student-code/

Classroom Climate

JHU is committed to creating a classroom environment that values the diversity of experiences and perspectives that all students bring. Everyone has the right to be treated with dignity and respect. Fostering an inclusive climate is important. Research and experience show that students who interact with peers who are different from themselves learn new things and experience tangible educational outcomes. At no time in this learning process should someone be singled out or treated unequally on the basis of any seen or unseen part of their identity. 
 
If you have concerns in this course about harassment, discrimination, or any unequal treatment, or if you seek accommodations or resources, please reach out to the course instructor directly. Reporting will never impact your course grade. You may also share concerns with your program chair, the Assistant Dean for Diversity and Inclusion, or the Office of Institutional Equity. In handling reports, people will protect your privacy as much as possible, but faculty and staff are required to officially report information for some cases (e.g. sexual harassment).

Course Auditing

When a student enrolls in an EP course with “audit” status, the student must reach an understanding with the instructor as to what is required to earn the “audit.” If the student does not meet those expectations, the instructor must notify the EP Registration Team [EP-Registration@exchange.johnshopkins.edu] in order for the student to be retroactively dropped or withdrawn from the course (depending on when the "audit" was requested and in accordance with EP registration deadlines). All lecture content will remain accessible to auditing students, but access to all other course material is left to the discretion of the instructor.