10 Dangerous Misconceptions About Vibe Coding: The Characteristics of Language Models
Haebom
Haebom's Archive
Leave a comment about IT, economy, and humanities.
Haebom's Archive
"Our company doesn't need developers anymore. We just need ChatGPT!"
Have you ever heard this? In the current AI craze, many people are portraying LLM (Large-Scale Language Model) as if it were a panacea. Especially those who tout an "AI revolution" and sell unproven solutions often try to hide LLM's limitations.
But what's the reality? LLMs are undoubtedly powerful tools. However, blindly relying on them without properly understanding their true nature can lead to serious problems, from security vulnerabilities to project failure.
In fact, after writing the article below, I received numerous contacts and publishing offers, and many other things happened. However, as I've written in the book and have said numerous times, Vibe Coding is practically impossible to teach in a lecture format. Workshops or guided tours might be possible, though.
I'll list down some common misconceptions and propaganda/agitation that I've heard from people who use services like Vibecoding or AI-powered coding.
Myth 1: "LLM-generated code always/sometimes works."
False beliefs
"Since it's code created by AI, it should work, right?" or "Sometimes it doesn't, but most of the time it works, right?"
Truth
The effectiveness of LLM-generated code is highly uncertain . Recent research suggests that more than 40% of AI-generated code contains security vulnerabilities. Just because code compiles or runs doesn't mean it "works."
It's like asking a five-year-old to prepare dinner. They may have the recipe memorized perfectly, but they don't fully understand the dangers of heat control or how to use a knife. While the result may be good, whether it's safe and wholesome food is a separate issue.
# LLM이 생성할 수 있는 "작동하는" 코드
while True:
pass # 무한 루프 - 컴파일도 되고 실행도 되지만...The right approach : LLM-generated code should always be treated as a draft . It must be reviewed, tested, and security vetted.
Myth 2: "An LLM can determine if your code is correct."
False beliefs
"That's right, since AI can verify the code."
Truth
LLM cannot determine the correctness of the code . This is also related to the Halting Problem, a fundamental limitation of computer science. Because LLM cannot know in advance the number of tokens it will generate, it cannot verify its output in advance.
LLM is a pattern recognition machine . It generates statistically plausible code, but it cannot determine whether the code is logically correct or whether it handles all edge cases.
Just as a parrot can mimic human speech, it cannot understand the meaning of the words. The same is true for an LLM.
The right approach : Code verification should be done by humans. Unit testing, integration testing, and code reviews are still essential.
Myth 3: "If it works as I expect, it's correct."
False beliefs
"I tested it and it came out just as I wanted. So, it's okay, right?"
Truth
This is one of the most dangerous misconceptions: your expectations themselves may be flawed .
# 사용자의 요청: "숫자를 두 배로 만드는 함수 만들어줘"
def double(x):
return x + x # 문자열 입력시 문제 발생!
double("hello") # "hellohello" - 기대와 다른 결과The above code works "as expected" for numbers, but it behaves unexpectedly for other inputs. It's like a child taking toothpaste when told to "brush your teeth." On the surface, it's following the instructions, but fundamentally, it's a misunderstanding.
The right approach : clearly define your specifications and write comprehensive tests that cover edge cases. Validate against the requirements document, not user expectations.
Myth 4: "LLM only uses existing libraries and APIs."
False beliefs
“Since it’s a library recommended by AI, it must actually exist, right?”
Truth
Absolutely not. This is called "AI hallucination," and LLM boldly suggests non-existent packages or APIs.
A recent study found that 21.7% of package names for open-source AI models were hallucinatory, and 5.2% of commercial models had non-existent packages.
What's more serious is that hackers exploit this. Attackers create and distribute malicious packages using names LLM conjures up. Developers then believe LLM's suggestion and install the malicious packages.
// LLM이 제안한 코드
import { parseData } from 'awesome-data-parser-2024'; // 존재하지 않는 패키지!
// 실제로는
npm install awesome-data-parser-2024
// Error: Package not foundThe Right Approach : Be sure to review the official documentation for all libraries and APIs suggested by LLM . Before installing a package, check its actual existence, download count, and most recent update date on npm, PyPI, and other platforms.
Myth 5: "LLM-generated code is secure."
False beliefs
"This code was created by cutting-edge AI, so there's no way there's a security issue."
Truth
Quite the opposite. A study by Backslash Security found that when generating code using simple prompts, all LLMs generated code containing at least four vulnerabilities from the Common Weakness Enumeration (CWE) Top 10. Specifically, the GPT-4o model contained vulnerabilities 90% of the time when using simple prompts.
Common security issues:
•
SQL injection : inserting user input directly into a query.
•
XSS (Cross-Site Scripting) : Missing input validation
•
Hardcoded passwords : Writing authentication information directly into the code.
•
Path Traversal Attack : Lack of File Path Verification
LLMs tend to by default omit input validation. Even when explicitly asked to "write secure code," they often only apply inconsistent or overly simplistic validation.
# LLM이 생성할 수 있는 취약한 코드
def get_user_data(user_id):
query = f"SELECT * FROM users WHERE id = {user_id}" # SQL 인젝션 위험!
return execute_query(query)
# user_id = "1 OR 1=1" 입력시 모든 사용자 정보 노출The right approach : LLM-generated code should be inspected with automated security scanning tools and undergo a code review by a security expert. Use a security checklist like the OWASP Top 10.
Myth #6: "LLMs think and reason."
False beliefs
"The AI is coming up with such smart answers, it must be thinking."
Truth
LLM doesn't think . It doesn't reason. It simply predicts the statistically most likely next word.
Borrowing a definition from philosopher Harry Frankfurt, some researchers describe LLM's output as "bullshit." LLM is indifferent to the truth of its output; true statements just happen to be true, and false statements just happen to be false.
Even OpenAI's latest inference models showed a 33% error rate for the o3 model and a 48% error rate for the o4-mini model when asked about public documents.
This is similar to a child listening to an adult's conversation and imitating it. They only learn the word patterns without understanding the context.
The right approach : View the LLM as a tool . It's not a thinking partner, but an advanced auto-completion system that quickly finds patterns. The final judgment and verification always rests with the human.
Myth 7: "LLMs remember what you said five minutes ago."
False beliefs
"I told you before, the AI will remember, right?"
Truth
LLMs have no long-term memory . Each conversation proceeds by rereading the entire context from the beginning. They have a limited amount of memory, called the context window, beyond which previous information is forgotten.
If the conversation gets long:
•
Start ignoring the initial instructions
•
"Forgets" previously defined variables or functions
•
Generate code with inconsistent style
It's like the main character in the movie '50 First Kisses' who resets his memory every morning.
# 대화 초반
"user_id를 정수로 처리해줘"
def get_user(user_id: int): ...
# 대화 후반 (맥락을 벗어나면)
"user_id를 문자열로 처리해줘" # 이전 지시사항과 모순!
def update_user(user_id: str): ...The right approach : Repeat key requirements and constraints with each prompt . Instead of trying to convey the entire project context all at once, break it down into smaller chunks with clear documentation.
Myth 8: "An LLM will do my best work."
False beliefs
“Since it’s an AI assistant, it’ll naturally do its best to help me, right?”
Truth
LLM has no goals . It neither wants to help you nor harm you. It simply reproduces the patterns in the training data.
This has important implications:
•
If you ask the wrong question, I will answer it honestly (without prompting you to ask the right question).
•
It doesn't warn you if there is a security vulnerability (if that's a pattern in the training data).
•
Even if there is a better approach, I may not suggest it.
The right approach : Treat LLMs like irresponsible interns . While talented, they can easily derail projects without supervision. Clear direction, constant review, and ultimate responsibility always lie with the individual.
Myth 9: "The LLM warns you about vague requirements."
False beliefs
"If my explanation is unclear, the AI will ask questions, right?"
Truth
LLMs attempt to fill in the blanks plausibly . Instead of saying "I don't know" when you're unsure, they make up plausible answers.
"AI illusions arise because of the probabilistic nature of AI models," explains a senior engineer at Microsoft. "They generate outputs based on statistical likelihood, not deterministic logic."
사용자: "데이터를 처리하는 함수 만들어줘"
LLM: (어떤 데이터? 어떻게 처리? 모르지만 일단 뭔가 만들자!)
def process_data(data): # 너무 일반적이고 실용적이지 않은 코드 생성
return [item for item in data]The right approach : Be specific and clear with your requirements . Be specific, such as, "Create a function that extracts only email addresses from a user input string and returns them as a list. Ignore any incorrect formatting."
Myth 10: "I'm good at asking for what I want."
False beliefs
"Even if I don't know how to program, if I tell the AI what I want, it will do it."
Truth
This is the most fundamental and often overlooked problem: Most people don't know exactly what they want.
The difficulty of software development lies not in the coding itself, but in clearly defining the problem and structuring the solution . An LLM cannot replace this process.
For example:
•
"Make a user-friendly interface" → What is user-friendly?
•
"Make me a fast algorithm" → How fast is fast?
•
"Make me safe" → Safe from what threats?
It's like telling a child to "play well." The child doesn't know what you mean by "play well."
The right approach : Conduct a requirements analysis before using an LLM . First, determine what, why, and how to build it. If you don't have programming experience, collaboration between domain experts and developers is still necessary.
The misconception that AI will take developers' jobs
An LLM is undoubtedly a revolutionary tool. It helps you write code faster, reduce repetitive tasks, and learn new technologies. But it's not a panacea.
Beware of those who claim, "If you have AI, you won't need developers." They 're selling an unproven dream . Companies that have actually implemented LLMs in production know that AI tools don't replace developers; they require a higher level of skill .
Developers in the LLM era:
•
Not the person who writes the code, but the person who reviews and verifies the code.
•
Not just an implementer, but someone who designs architecture.
•
Not just someone who knows technology, but someone who understands business requirements.
I often say in class: Treat your LLM like a child. Like a talented child, they possess great potential, but without proper guidance and supervision, they can go astray. Don't blindly trust them, but don't ignore them either. Maintaining critical thinking and utilizing them wisely is our challenge in the AI era.
Made with Slashpage