A buffer overflow occurs when a program or process tries to store more data in a buffer (temporary data storage area) than it was intended to hold. Since buffers are created to contain a finite amount of data, the extra information - which has to go somewhere - can overflow into adjacent buffers, corrupting or overwriting the valid data held in them. -techtarget
The attacker will need to be able to identify a buffer overflow vulnerability in some program and understand how that buffer will be stored in the process memory, and know what other adjacent memory items he could write to.
To do this, could use a technique called fuzzing.
Fuzzing involves sending malformed strings into application (and web) input and watching for unexpected crashes. There are lots of interesting tutorials as to how to do this. Many times after finding this malformed input, you would then use assembly to figure out how to deliver shellcode (or how to deliver the exploit).
Fuzzing does have practical usage in software development, but it is also a tool used by hackers to find vulnerabilities in applications.
Many fuzzing tools: ComRaider (activeX), see /pentest/fuzzers
Software security is closely related to software quality and reliability.
Defensive programming: intended to ensure the continuing function of a piece of software in spite of unforseeable usage of said software. Sometimes this is called “Secure Programming”. Never make any assumptions about code
Most programmers focus on solving a problem, (the algorithm involved) rather than considering every point of failure. Often make assumptions about the input and the environment it executes in. Defensive programmers should:
Software security has to be implemented in design phase of the development rather than as an afterthought.
One of the most common failures in software security.
The input is not always explicitly known. Making assumptions about the input can be bad. You should always handle the input cautiously to make sure that it isn’t invalid.
Two key areas of concern for input are:
Don’t make any assumptions, validate everything.
Do the inputs conform to what you are expecting? ie. If you are expecting a filename, does the input follow the typical filename pattern?
Injection attack: Essentially this is where an attacker provides non-expected input in order to do something that they shouldn’t. It influences the control or execution of a program. Commonly seen in PHP or other webscripts
Expected input = expected results, but what if the attacker injects a command, i.e. xxx; echo success; ls -l finger *