645.755.81 - Methods in Human-System Performance Measurement and Analysis

Expanded Course Description

Systems are designed, built, and used by humans. Their purpose is to help people meet their goals and perform their tasks. This course examines human performance measurement (HPM) used in determining whether human-system requirements are met and if systems designs provide optimal human-system performance. The objective is to provide students with the knowledge of HPM study designs and data collection tools and methods, as well as the Institutional Review Board (IRB) process and procedures (protocols) required to execute studies with human subjects.  Metrics, measurements, tools providing for performance and preference measures will be detailed enough to provide insights for system architectures supporting data collection.  HSI heuristic evaluations will be discussed.   

A second objective is to provide students an understanding of HPM in the context system design, environment, safety, occupational health; training and maintenance. Students will be exposed to psychophysics, cognitive measurement, assessment of psychomotor skills and controls, use of workload and task analysis in modeling & simulation, and measuring and offsetting overload and fatigue.  An emphasis will be placed on system testing of the human element, including Reliability, Maintainability, Availability (RM&A); and effectiveness and efficiency (e.g.,  performance, preference, usability, utility, learn-ability, maintainability, accessibility, confidence, responsiveness, and other "-ilities“).  Case studies revealing the impact of usability on human performance will be presented.   

Please see Instructor Biographies for information about the Instructors who developed and are teaching this course.

Course Structure

The course materials are divided into modules. The modules can be accessed by clicking “Modules” on the course menu. A module will generally have four sections: the readings, lectures (videos and PDF files), assignments, and discussions. Students are encouraged to preview all sections of the module before starting. Most modules run for a period of seven (7) days; exceptions are noted in the Course Schedule. Students are encouraged to do the reading and then view the lectures during the first 3-4 days and participate in discussions afterwards, for example during the last 3-4 days.  Assignments are due the last (seventh) day of the module.  Students should regularly check the Announcements and the Calendar for assignment due dates. 

Course Goals

The goal of this course is to lay the foundations for understanding Human-Systems Performance Measurement as a science for assessing the human performance component of complex systems.

By the end of the course, students will be able to: 

• Appreciate human performance assessment within Human System Integration (HSI) domains and as a component of systems test 
• Identify and describe the characteristics and responsibilities of Human Performance Measurement (HPM) teams and team members 
• Realize the basic foundations of experimental design 
• Apprehend how to select appropriate human performance metrics 
• Understand the principles of test and evaluation of systems employing human operators
• Plan for measuring humans as a part of total system design 
• Develop test plan requirements that include HPM 
• Set reasonable expectations of a test and evaluation team 

Textbooks

Human Performance, Workload, and Situational Awareness Measures Handbook, Second Edition.
Author: Valerie J. Gawron
Edition/Copyright: 2008
Publisher: Taylor & Francis Group, LLC
ISBN-13:  978-1-4200-6449-0 

Required Software

• A current web browser (e.g., Firefox or Chrome)
• Microsoft Office Suite, particularly:
  
  • PowerPoint
    
  • Word
    
  • Excel
    
• A PDF viewer, such as Adobe Acrobat Reader
  

Student Coursework Requirements

It is expected that each class will take approximately 6–8 hours per week to complete. Here is an approximate breakdown:

• Reading the assigned book and/or outside materials (approximately 2–3 hours per week)
• Watching module lectures (including reading the lecture slides) (approximately 2 1/2 hours per week)
• Completing the Assignments and Discussions (approximately 2-3 hours per week)

A course project will be assigned the first week the course. The entire semester and most assignments and Discussion topics will be focused on progressing the project toward the final presentation.

This course will consist of four basic student requirements:

1. Class Preparation and Participation in Discussions (20% of Final Grade Calculation)
2. Assignments (25% of Final Grade Calculation)
3. Midterm Brief (25% of Grade Calculation)
4. Final Project Presentation (30% of Grade Calculation)

Each of these is described further below.

1. Class Preparation and Participation in Discussions

Each student is responsible for carefully reading all assigned material and being prepared for discussion. Some readings are from the course text. Additional readings will be assigned to supplement the text.

The instructors will host and monitor class discussions and will respond to discussions as they are posted.

Evaluation of this requirement is based on collaboration quality rather than quantity, and is weighted as follows:

• Timeliness (50%)
• Critical Thinking (50%)

Class Preparation and Participation in Discussions is graded as follows:

• 100–90 = A
  
  • Timeliness: Regularly participates; all required postings; early in discussion; throughout the discussion.
  • Critical Thinking: Rich in content; full of thoughts, insight, and analysis.
• 89–80 = B
  
  • Timeliness: Frequently participates; all required postings; some not in time for others to read and respond.
  • Critical Thinking: Substantial information; thought, insight, and analysis has taken place.
• 79–70 = C
  
  • Timeliness: Infrequently participates; all required postings; most at the last minute without allowing for response time.
  • Critical Thinking: Generally competent; information is thin and commonplace.
• < 70 = F
  
  • Timeliness: Rarely participates; some, or all required postings missing.
  • Critical Thinking: Rudimentary and superficial; no analysis or insight is displayed.
    

2. Assignments

All assignments are due according to the dates in the Calendar. Late submissions will be reduced by one letter grade for each week late (no exceptions without prior coordination with the instructors).

Students are expected to use APA Style for the formatting of citations and references.

We expect that all Assignments will be submitted with proper spelling, punctuation, grammar, and syntax. Minor typographical errors may be noted for correction, but will not generally result in loss of points. Egregious violations of the rules of the English language, on the other hand, will be noticed and may affect the grade, depending on the circumstance. Consistently poor performance in spelling or grammar may be taken as an indication of poor written communication skills that may detract from Assignment grades. 

Assignments will include a mix of qualitative assignments (e.g. literature research, reviews, recordings, and writing), quantitative problem sets, and case study updates. Each answer should have the question/problem statement, assumptions, computations, and conclusions/discussion delineated. All Figures and Tables should be captioned and labeled appropriately. Include keys with figures.

Assignments are evaluated using the criteria supplied with the Assignment.  The following are expected across all Assignments:

1. Each part of the question/instructions are answered/addressed, respectively.
2. Assumptions and hypotheses are clearly stated.
3. Quantitative answers:
   • Intermediate derivations and calculations are provided.
   • Answer is technically correct and is clearly indicated.
   • Answer precision and units are appropriate.
4. Qualitative answers:  
   • Writing quality meets or exceeds accepted graduate-level English and scholarship standards. (Assignments will be graded on spelling, grammar, and style.)
   • Answer is technically correct and is clearly indicated.
5. Rationale for answer is provided.
6. Examples are included to illustrate rationale. (If a student does not have direct experience related to a particular question, then the student is to provide analogies versus examples.)
7. Demonstrates the “Big Four” components:
   • Consideration of the user(s) (e.g., operator, trainer and maintainer)
   • Consideration of context (e.g., culture and environment)
   • Consideration of worst case(s) in addition to typical case(s)
   • Proper APA Style citations and references

A grading rubric for Assignments is as follows:

Grade	Quantitative Portions	Qualitative Portions
100–90 = A	All parts of question are addressed; All assumptions are clearly stated; All intermediate derivations and calculations are provided; Answer is technically correct and is clearly indicated; Answer precision and units are appropriate.	All parts of question are addressed; Writing Quality/ Rationale/ Examples/ Outside References are rich in content and full of thought, insight, and analysis.
89–80 = B	All parts of question are addressed; All assumptions are clearly stated; some intermediate derivations and calculations are provided; Answer is technically correct and is indicated; Answer precision and units are appropriate.	All parts of the question are addressed; Writing Quality/ Rationale/ Examples/ Outside References contain substantial information; thought, insight, and analysis has taken place.
79–70 = C	Most parts of question are addressed; Assumptions are partially stated; Few intermediate derivations and calculations are provided; Answer is not technically correct but is indicated; Answer precision and units are indicated but inappropriate.	Majority of parts of the question are addressed; Writing Quality/ Rationale/ Examples/ Outside References are generally competent; information is thin and commonplace.
< 70 = F	Some parts of the question are addressed; Assumptions are not stated; Intermediate derivations and calculations are not provided; The answer is incorrect or missing; The answer precision and units are inappropriate or missing.	Some parts of the question are addressed; Writing Quality/ Rationale/ Examples/ Outside References are rudimentary and superficial; no analysis or insight is displayed.

3. Midterm Brief

Instructions for the midterm will be available in the 7th week, March 6th. Students will have two weeks to complete the midterm (not counting Spring Break), and it will be due March 31st. Students may use the course text and class lectures and notes to complete the midterm.

Expectations are the same as those provided in other assignments (see #2 above).

4. Final Project Presentation

Instructions for presenting the course project as the class final will be available after the midterm on March 29th. The next-to-the-last week will be devoted to preparing and recording the course project.  The final presentation will be recorded, with the link and the brief submitted.  Students may arrange to record their presentation with the instructor by emailing the instructor two weeks in advance.  Students will have one-half hour to complete the brief (including 5 mins of questions). Students may use the course text and class lectures and notes to complete the brief. The final version of the course project, including brief slides, video, and associated documents must be submitted by Friday, May 13, 2023.

Expectations are the same as those provided in other assignments (see #2 above).

The project is evaluated by the following grading elements:

1. Student preparation and participation 
2. Student technical understanding of the project topics
3. Team preparation and participation 
4. Team technical understanding of the project topic 

The Class Project is graded as follows:

Grade	Student / Team Preparation & Participation	Student / Team Technical Understanding
100–90 = A	—Individual / team roles and responsibilities are well defined and understood —Individual / team work product(s) agreed to, well prepared and available to all team members/ instructors	—Rich in content —Full of thought, insight, and analysis
89–80 = B	—Individual/ team roles and responsibilities well defined and understood —Individual/ team work product(s) agreed to and prepared	—Substantial information —Thought, insight, and analysis has taken place
79–70 = C	—Individual/ team roles and responsibilities agreed to —Individual/ team work product(s) prepared	—Generally competent —Information is thin and commonplace
< 70 = F	—Individual/ team roles and responsibilities not well understood —Individual/ team work product(s) partially prepared	—Rudimentary and superficial —No analysis or insight displayed

Grading Policy

Grades will be posted 1-2 weeks after assignment due dates.

A grade of A indicates achievement of consistent excellence and distinction throughout the course—that is, conspicuous excellence in all aspects of assignments and discussion in every week. 

A grade of B indicates work that meets all course requirements on a level appropriate for graduate academic work. These criteria apply to both undergraduates and graduate students taking the course. 

EP uses a +/- grading system (see “Grading System”, Graduate Programs catalog, p. 10).

Percent	Letter Grade	Meaning
100-97	= A+	Excellent
96-93	= A
92-90	= A−
89-87	= B+	Good
86-83	= B
82-80	= B−
79-77	= C+	Unsatisfactory
76-73	= C
72-70	= C−
69-67	= D+	Failure
66-63	= D
<63	= F
	I	Incomplete
	W	Official Withdrawal
	AU	Audit
	R	Replaced/Retaken

Final grades will be determined by the weighting described in the Student Coursework Requirements section.

Academic Policies

Deadlines for Adding, Dropping and Withdrawing from Courses

Students may add a course up to one week after the start of the term for that particular course. Students may drop courses according to the drop deadlines outlined in the EP academic calendar (https://ep.jhu.edu/student-services/academic-calendar/). Between the 6th week of the class and prior to the final withdrawal deadline, a student may withdraw from a course with a W on their academic record. A record of the course will remain on the academic record with a W appearing in the grade column to indicate that the student registered and withdrew from the course.

Academic Misconduct Policy

All students are required to read, know, and comply with the Johns Hopkins University Krieger School of Arts and Sciences (KSAS) / Whiting School of Engineering (WSE) Procedures for Handling Allegations of Misconduct by Full-Time and Part-Time Graduate Students.

This policy prohibits academic misconduct, including but not limited to the following: cheating or facilitating cheating; plagiarism; reuse of assignments; unauthorized collaboration; alteration of graded assignments; and unfair competition. Course materials (old assignments, texts, or examinations, etc.) should not be shared unless authorized by the course instructor. Any questions related to this policy should be directed to EP’s academic integrity officer at ep-academic-integrity@jhu.edu.

Students with Disabilities - Accommodations and Accessibility

Johns Hopkins University values diversity and inclusion. We are committed to providing welcoming, equitable, and accessible educational experiences for all students. Students with disabilities (including those with psychological conditions, medical conditions and temporary disabilities) can request accommodations for this course by providing an Accommodation Letter issued by Student Disability Services (SDS). Please request accommodations for this course as early as possible to provide time for effective communication and arrangements.

For further information or to start the process of requesting accommodations, please contact Student Disability Services at Engineering for Professionals, ep-disability-svcs@jhu.edu.

Student Conduct Code

The fundamental purpose of the JHU regulation of student conduct is to promote and to protect the health, safety, welfare, property, and rights of all members of the University community as well as to promote the orderly operation of the University and to safeguard its property and facilities. As members of the University community, students accept certain responsibilities which support the educational mission and create an environment in which all students are afforded the same opportunity to succeed academically. 

For a full description of the code please visit the following website: https://studentaffairs.jhu.edu/policies-guidelines/student-code/

Classroom Climate

JHU is committed to creating a classroom environment that values the diversity of experiences and perspectives that all students bring. Everyone has the right to be treated with dignity and respect. Fostering an inclusive climate is important. Research and experience show that students who interact with peers who are different from themselves learn new things and experience tangible educational outcomes. At no time in this learning process should someone be singled out or treated unequally on the basis of any seen or unseen part of their identity. 
 
If you have concerns in this course about harassment, discrimination, or any unequal treatment, or if you seek accommodations or resources, please reach out to the course instructor directly. Reporting will never impact your course grade. You may also share concerns with your program chair, the Assistant Dean for Diversity and Inclusion, or the Office of Institutional Equity. In handling reports, people will protect your privacy as much as possible, but faculty and staff are required to officially report information for some cases (e.g. sexual harassment).

Course Auditing

When a student enrolls in an EP course with “audit” status, the student must reach an understanding with the instructor as to what is required to earn the “audit.” If the student does not meet those expectations, the instructor must notify the EP Registration Team [EP-Registration@exchange.johnshopkins.edu] in order for the student to be retroactively dropped or withdrawn from the course (depending on when the "audit" was requested and in accordance with EP registration deadlines). All lecture content will remain accessible to auditing students, but access to all other course material is left to the discretion of the instructor.