To read about more details, please refer to the arxiv paper: https://arxiv.org/abs/2305.13272
For more related informationn, find the CLASS slides here.
To read about the follow-up work, refer to Pedagogical Alignment of Large Language Models

We aim to develop more effective educational tutoring chatbots (Spock) that help students deepen their understanding of academic subjects. To train the chatbot, we create a synthetic dataset of mock conversations between a student and a tutor based on learning science principles like scaffolding. We employ a specialized prompt to generate these mock conversations using OpenAI's GPT-4 APIs. Leveraging these conversational datasets, we supervise fine-tune the chatbots from Mistral, Zephyr, and Vicuna using FastChat and then rlhf fine-tune the chatbots using TRL.

To use the models, first install the fastchat library into the root dir, and then follow the steps here:

1. Update the conversation.py from our repository in the FastChat folder.
2. Update the inference.py from our repository in the FastChat folder.
3. Use the apply_delta.py on Spock-Bio-Llama-Diff to get actual model weights.
   • Example: python3 -m fastchat.model.apply_delta --base decapoda-research/llama-13b-hf --target tutorbot_spock_vicuna_prompt_v3 --delta luffycodes/tutorbot-spock-bio-llama-diff
   • Also, please put vicuna in the target model name since conversation.py and inference.py check if vicuna is a substring in a model name and change conversation starter and inference prompts respectively. Note we modify vicuna prompts so you would not able to able to use original vicuna models unless you revert back changes to conversation.py and inference.py.
4. Build a biology index with OpenStax Biology 2e textbook. Put the generated os_bio_2e_index.faiss and the openstax_biology_2e.csv in same folder as inference.py i.e. FastChat/fastchat folder.

For easier access of the models, download them directly from Hugging Face.
SFT Models:

• mistral-7b-instruct-v0.2-class-bio-tutor-sft
• zephyr-7b-beta-class-bio-tutor-sft
• vicuna-7b-v1.5-class-bio-tutor-sft

SFT + DPO Models:

• mistral-7b-instruct-v0.2-class-bio-tutor-dpo
• zephyr-7b-beta-class-bio-tutor-dpo
• vicuna-7b-v1.5-class-bio-tutor-dpo

1. Run the mock_con_GPTx_prompt_v3.py
   • It uses conversation prompt v3
2. Remember to put openai.organization and openai.api_key in the file
3. To create a scaffolding dataset, use prompts in folder

Please refer to a more detailed readme.md inside the rlhf folder for training and evaluating the models.

If you use this work, please cite:

• CLASS Meet SPOCK: An Education Tutoring Chatbot based on Learning Science Principles
  
• Pedagogical Alignment of Large Language Models
  

@misc{sonkar2023class,
      title={CLASS Meet SPOCK: An Education Tutoring Chatbot based on Learning Science Principles}, 
      author={Shashank Sonkar and Lucy Liu and Debshila Basu Mallick and Richard G. Baraniuk},
      year={2023},
      eprint={2305.13272},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@misc{sonkar2024pedagogical,
      title={Pedagogical Alignment of Large Language Models}, 
      author={Shashank Sonkar and Kangqi Ni and Sapana Chaudhary and Richard G. Baraniuk},
      year={2024},
      eprint={2402.05000},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}