Syllabus

Basic information

  • 36-627/727 Modern Experimental Design
  • Spring 2026, mini 4
  • Monday and Wednesday, 9:30–10:50am
  • Instructor: Alex Reinhart
  • TA: Xander Brick
  • Office hours: TBD

Course description

Designed experiments are crucial to draw causal conclusions with minimum expense and maximum precision. This course introduces the basic principles and theory of experimental design, including randomized designs, blocking, analysis of covariance, factorial designs, and power analysis, with an emphasis on recent techniques often applied to the online experiments frequently used by tech companies. We will emphasize the importance of critical thinking about the goals and context of an experiment to choose the best design, and practice these skills through a course project.

This course is primarily for students in the Master of Science in Applied Data Science program and the Statistics PhD program. Students in this course should have prior experience with probability, mathematical statistics (such as estimation, hypothesis testing, and confidence intervals), linear algebra, and regression analysis. Coursework will primarily use R for the analysis of experimental data.

This course is cross-listed as 36-627 and 36-727. MADS and other master’s students should enroll in 627; PhD students should enroll in 727. There is no difference between the sections—the difference is purely for bureaucratic reasons.

Learning objectives

Students will learn

  • to recognize the goals, constraints, and limitations involved in designing an experiment for a particular problem,
  • to design an appropriate experiment to answer specific substantive questions,
  • to analyze experimental data using a variety of statistical methods in R,
  • to use simulations and power analyses to explore the weaknesses of a given design, and
  • to communicate design decisions, analyses, and substantive results in written reports.

Books and references

We will refer to several books in this course:

  • Giesbrecht and Gumpertz (2004) (freely available through CMU library access to Wiley)
  • Imbens and Rubin (2015)

Coursework

There will be three types of work in this course:

Readings

I will assign readings to be done before class each week. These readings may be from the course notes or from books or papers I provide. To encourage reading and discussion, I will post several questions on Canvas as a “quiz” to be completed before class. This quiz is for completion credit only.

Assignments

There will generally be a homework assignment due each week. Assignments may involve analysis of experimental data, derivations, simulation tasks, descriptions of appropriate experimental designs for challenging situations, and so on.

Some assignments may be in-class assignments, meaning they are activities completed during class and due at the end of the class period or the beginning of the next. For example, you may be asked to derive a result or work out an interesting experimental design in class. Other assignments may be reading assignments for a specific class, requiring you to read a paper or chapter before class and complete some task related to it.

Project

There will be a course project in lieu of a final exam. The project will involve designing and running an experiment in a challenging situation with resource constraints, and writing a report describing the design, analysis, and conclusions.

The project will be due in several pieces. A proposed design and analysis will be due midway through the course, and will be returned with comments and suggestions which should be incorporated in the final analysis and report. Further details will be given in class.

Submission

Homework may be submitted handwritten (scanned into PDF), as PDF files from LaTeX, or as PDF output from R Markdown or Quarto. All analyses, simulations, and reports (including the project) must be submitted as reproducible R Markdown or Quarto files.

For homework and the project, you will have three “grace days” you can use throughout the semester. Each time you use a grace day for an assignment, you get 24 hours extra to submit the assignment. You do not need any excuse to use grace days. Once you have used all three grace days, late work will not be accepted. To submit an assignment using a grace day, email it directly to the instructor.

This system is meant to allow you flexibility, so that ordinary problems (minor illness, forgot a deadline, had to finish another class’s big assignment, traveled to an event) don’t harm you, and so you do not need my permission to handle unexpected problems. If you experience a serious emergency that prevents you from completing work for a longer time, contact me so we can make arrangements.

Late reading assignments will not be accepted, since reading assignments are intended to prepare you for a specific day of class.

Attendance and participation

Class attendance and participation is essential. If there’s any one message to be learned from pedagogical research, it’s that listening passively to a lecture is not a good way to learn how to think about complicated problems. As a result, we will use much of our class time for demonstrations and activities, such as

  • solving conceptual problems about experimental designs
  • analyzing real experimental data using R
  • running simulations to validate tests or diagnostics
  • examining data case studies to determine the appropriate designs and analyses to solve real problems

You are expected to attend class and participate in these activities. Many of the activities will be expanded upon in homework assignments and submitted for homeork credit.

If you cannot attend a class for any reason, please let me know as far in advance as is possible. Class sessions are not recorded, and remote attendance of in-person classes is not possible.

Mobile devices

You should bring your laptop to class, as some activities will involve using R for data analysis.

However, there is significant evidence that using your laptops for other tasks in class—homework assignments for other courses, placing sports bets, ordering hats for your cockatiel—has a strong negative effect on your learning. It also distracts the students around you. Furthermore, there is evidence that students who say and even believe they are merely taking notes are frequently distracted by other things on their laptops, and often are unaware of just how distracted they are.

Consequently, I ask that you do not use laptops or other mobile devices for any other purpose except class activities and approved accommodations.

Grading

Final grades will be based on:

Item Percent
Reading quizzes 10%
Homework 60%
Project 30%

Academic integrity

Collaboration

Discussing homework and projects with your classmates is allowed and encouraged, and helping explain ideas to each other is a core part of the academic experience. But it is important that every student get practice working on their own. This means that all the work you turn in must be your own. You must devise and write your own code, generate your own graphics, and write your own solutions and reports.

Outside sources

You may use external sources (books, websites, papers) to

  • Look up R documentation, find useful packages, find explanations for error messages, or remind yourself about the functions to fit some model,
  • Find reference materials on statistical methods,
  • Clarify material from the course notes or examples.

But external sources must be used to support your work, not to obtain your work. You may not use them to copy code, text, or graphics without attribution. You may not use any prior course’s or textbook’s homework solutions in any way. This prohibition applies even to students who are re-taking the course. Do not copy old solutions (in whole or in part), and do not “consult” or read them. Doing any of that is cheating, making any feedback you get meaningless and any evaluation based on that assignment unfair.

If you do use any material from other sources, you must clearly mark its source. Text taken from other sources must be in quotation marks with citations; figures from other sources need a caption indicating the source; and code from other sources must have a comment indicating the source. We must be able to determine who wrote any material you submit, and you must not falsely imply that you completed work actually done by others.

Generative AI

Some of you may be tempted to use generative AI tools like ChatGPT, Gemini, GitHub Copilot, or Claude to complete some of your work in this course. My policy towards these tools depends on the type of the assignment, based on their learning goals:

  • Reading questions: Their purpose is to encourage you to think about the reading and to elicit your thoughts and questions. You must do the thinking, not generative AI. If you are not able to explain your answers orally when asked, you may not receive credit for completing the reading questions.

  • Homework: Homework enables your reflection on the course concepts and helps you recognize when you do not understand something. Using generative AI would defeat the purpose, so generative AI is not permitted.

  • Projects: Projects help you practice experimental design, data analysis, writing, and coding skills. Some of these skills can be assisted by generative AI: for example, it can help debug code or polish text, and can be good at this. You may consult generative AI tools, but you must still take intellectual responsibility for your code, analysis, and writing, and be able to explain and defend every decision. You may not use generative AI to come up with analysis or interpretation for you, only to augment your own work. If you use a generative AI tool, you must disclose your use in your submission and explain what you used it for.

    However, generative AI tools are likely not as good as you think they are. They write grammatically correct text, but they do not write about data analysis like humans do, and their style is very distinct (even if prompted to write in a certain way). See Reinhart et al. (2025) and DeLuca et al. (2025). If you use generative AI to help write reports, expect to make extensive revisions manually or risk receiving a lower grade. And if you are unable to orally explain your design, decision-making, and results when asked, you may not receive credit for the submission.

I will consider the use of generative AI tools in violation of this policy to be “unauthorized assistance”, as defined in the University Policy on Academic Integrity.

Penalties

Please talk to me if you have any questions about this policy. Any form of cheating, unauthorized assistance, or plagiarism is grounds for sanctions to be determined by the instructor, including grade penalties or course failure. Students taking the course pass/fail may have this status revoked. I am also obliged in these situations to report the incident to your academic program and the appropriate University authorities. Please refer to the University Policy on Academic Integrity.

Accommodations for students with disabilities

If you have a disability and have an accommodations letter from the Disability Resources office, I encourage you to discuss your accommodations and needs with me as early in the semester as possible. I will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, I encourage you to contact them at .

Diversity and inclusion

We must treat every individual with respect. We are diverse in many ways, and this diversity is fundamental to building and maintaining an equitable and inclusive campus community. Diversity can refer to multiple ways that we identify ourselves, including but not limited to race, color, national origin, language, sex, disability, age, sexual orientation, gender identity, religion, creed, ancestry, belief, veteran status, or genetic information. Each of these diverse identities, along with many others not mentioned here, shape the perspectives our students, faculty, and staff bring to our campus. We, at CMU, will work to promote diversity, equity and inclusion not only because diversity fuels excellence and innovation, but because we want to pursue justice. We acknowledge our imperfections while we also fully commit to the work, inside and outside of our classrooms, of building and sustaining a campus community that increasingly embraces these core values.

Each of us is responsible for creating a safer, more inclusive environment.

Unfortunately, incidents of bias or discrimination do occur, whether intentional or unintentional. They contribute to creating an unwelcoming environment for individuals and groups at the university. Therefore, the university encourages anyone who experiences or observes unfair or hostile treatment on the basis of identity to speak out for justice and support, within the moment of the incident or after the incident has passed. Anyone can share these experiences using the following resources:

All reports will be documented and deliberated to determine if there should be any following actions. Regardless of incident type, the university will use all shared experiences to transform our campus climate to be more equitable and just.

Wellness

All of us benefit from support during times of struggle. There are many helpful resources available on campus and an important part of the college experience is learning how to ask for help. Asking for support sooner rather than later is almost always helpful.

If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support. Counseling and Psychological Services (CaPS) is here to help: call 412-268-2922 or visit their website. Consider reaching out to a friend, faculty or family member you trust for help getting connected to the support that can help.