Image-based modeling and rendering is currently one of the most challenging topics in Computer Vision and Photogrammetry. The key issue here is building a set of dense correspondence points between two images, namely dense matching or stereo matching. Among all dense matching algorithms, Semi-Global Matching (SGM) is arguably one of the most promising algorithms for real-time stereo vision. Compared with other global matching algorithms, SGM aggregates matching cost from several (eight or sixteen) directions rather than only the epipolar line using dynamic programming approach. Thus, SGM eliminates the classical “streaking problem” and greatly improves the accuracy and efficiency. In this paper, we aim at further improvement of SGM about its accuracy without increasing the computational cost. We propose setting the penalty parameters adaptively according to image edges extracted by edge detectors. We have carried out experiments on the standard Middlebury stereo dataset and evaluated the performance of our modified method with the ground truth. The results have shown a noticeable accuracy improvement compared with the results using fixed penalty parameters while the runtime computational cost was not increased.