Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., ... & Xie, X. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1-45

Wang, L., Chen, X., Deng, X., Wen, H., You, M., Liu, W., ... & Li, J. (2024). Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. npj Digital Medicine, 7(1), 41. Consistency and Reliability metrics test. GPT-4-Web : ChatGPT Consider they just compare with ChatGPT and GPT-API only focus on prompt technics without considering the system prompt.

Shen, L., Tan, W., Zheng, B., & Khashabi, D. (2023). Flatness-aware prompt selection improves accuracy and sample efficiency. arXiv preprint arXiv:2305.10713. An Theoretical approach to measure the effectiveness of prompt, "prompt flatness" Low citation, but interesting approach. Our results indicate that prompts with higher flatness generally lead to better accuracy. → Just for the ideation, but not be directly applicable to us for prompt engineering evaluation and test.

Radcliffe, T., Lockhart, E., & Wetherington, J. (2024). Automated prompt engineering for semantic vulnerabilities in large language models. The findings not only highlight the urgent need for more advanced security measures but also offer valuable insights into how prompt design and model refinement can mitigate these vulnerabilities.

Sun, H., Hüyük, A., & van der Schaar, M. (2023). Query-dependent prompt evaluation and optimization with offline inverse RL. In The Twelfth International Conference on Learning Representations. Identify the overlooked query-dependent prompt optimization objective and its challenges, and introduce Offline Inverse Reinforcement Learning to integrate rich human expertise as a systematic approach. (Prompt-OIRL) Jin, M., Yu, Q., Shu, D., Zhao, H., Hua, W., Meng, Y., ... & Du, M. (2024). The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.04925. Increasing Reasoning steps leads increasing in the effectiveness of CoT. But it depends on the complexity of task. Nayab, S., Rossolini, G., Buttazzo, G., Manes, N., & Giacomelli, F. (2024). Concise thoughts: Impact of output length on llm reasoning and cost. arXiv preprint arXiv:2407.19825.

Shubham Vatsal & Harsh Dubey(2024) Feature: Survey on Overall Prompt Engineering Techniques (Basic to latest) 각 NLP Task 별 적절한 Prompt engineering 기법들 정리

Paper Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT’s Customizability, Masaru Yamada(2023)

Entity 추출 활용 요약 프롬프팅 Chain of Density (CoD) prompt Structured (by The_Horse_Shiterer) Antropic Claud method prompt chaining Fabric (by Jonathan Dunn) Paper Prompt Chaining or Stepwise Prompt? Refinement in Text Summarization, Shichao Sun et al. (2024) Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method, Yiming Wang, Zhuosheng Zhang, Rui Wang (2023)

sLM Paper 1) Exploring Small Language Models with Prompt-Learning Paradigm for Efficient Domain-Specific Text Classification : 2) Knowledge Graph Structure as Prompt: Improving Small Language Models Capabilities for Knowledge-based Causal Discovery :https://arxiv.org/abs/2407.18752 3) Teaching Small Language Models to Reason for Knowledge-Intensive Multi-Hop Question Answering : https://aclanthology.org/2024.findings-acl.464.pdf 4) VersusDebias: Universal Zero-Shot Debiasing for Text-to-Image Models via SLM-Based Prompt Engineering and Generative Adversary :https://arxiv.org/abs/2407.19524 5) Effectively Prompting Small-sized Language Models for Cross-lingual Tasks via Winning Tickets : https://arxiv.org/abs/2404.01242 6-1) Orca 2: Teaching Small Language Models How to Reason : 6-2) 관련 정리 블로그 : https://chanmuzi.tistory.com/433 7-1) Teaching Small Language Models to Reason : https://arxiv.org/abs/2212.08410?source=post_page-----9107d9881075-------------------------------- 7-2) 관련 정리 블로그 글 : https://cobusgreyling.medium.com/teaching-small-language-models-to-reason-9107d9881075 8) It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners : h 9) Small Language Model Can Self-Correct : https://arxiv.org/abs/2401.07301

sLLM 중 하나인 Upstage의 Solar 모델 관련 문서 Solar에 대한 Introduction과 api를 받아와서 모델 설정하는 방법이 상세히 설명되어있습니다. Tutorials 탭의 "Create your own chatbot" 항목은 Multiple turns(Chat 형식) 구현 예제 APIs 탭은 Upstage의 다양한 models 관련 설명 ( function calling - solar-1-mini-chat, document-parse, etc) Upstage의 cookbook(Solar-Fullstack-LLM-101) Solar Model에 대한 전반적인 사용법(mainly LangChain 기반 RAG) RAG 결합 상황에서의 Solar 모델에 대한 prompting 연습하기 좋은 실습 예제들