Athol Fitzgibbons, Adam Smith's System of Liberty, Wealth, and Virtue, New York, Oxford University Press, 1995.
Far from thinking that commerce could solve moral problems, Smith subscribed to that part of the Hobbesian philosophy which identified unrestrained human beings with wild animals, with an inclination 'to extort all they can from their inferiors' (JA 23), and a 'love of dominion and authority over others' (JA 187). But unlike Hobbes, who had thought that human self-love and beastliness could be circumscribed only by an authoritarian state, and unlike Hume, who thought that the solution was commerce and a humane social consensus, Smith argued that each individual had an innate tendency to respect the rules of natural justice. 
The virtues of prudence, justice and beneficence, have no tendency to produce any but the most agreeable effects. (TMS 264) 
A liberal state could safely maintain a standing army:
Men of republican principles have been jealous of a standing army as dangerous to liberty... But where the sovereign himself is the general, and the principle nobility and gentry of the country the chief officers of the army; where the military force is placed under the command of those who have the greatest interest in the support of the civil authority, because they have themselves the greatest share of that authority, a standing army can never be dangerous to liberty. On the contrary, it may in some cases by favourable to liberty. (WN 706-7) 
The libertarian view that Smith 'erected a stupendous palace upon the granite of self-interest' (Stigler, in Skinner and Wilson: 237), trades on a confusion between the two terms. Smith did not regard self-love as the foundation of society; or in modern words the capitalist system was not originally meant to be built on greed. Mandeville had argued, in his Fable of the Bees, that economic life presupposed vanity, greed, and other forms of self-love, from which Mandeville had concluded that self-love was the great prerequisite for civilization. Although that position is often attributed to Smith, he retorted that Mandeville's 'licentious' system was marred by a basic error, despite its recognition of a fundamental truth. Mandeville had not understood that a particular degree of virtue was appropriate to each action. 'It is the great fallacy of Dr Mandeville's book to represent every passion as wholly vicious... in any degree and in any direction; (TMS 312). Smith's argument was that a whole range of actions, including ambition, enterprise, industry, frugality, and abstinence, expressed an inferior virtue or drew on mixed motives. A combination of high an low motives was usually appropriate, and the usual complaint that other people were selfish meant only that there was too much self-love and not enough benevolence in a particular instance:
The cause of this [complaint], however, is not that self-love can never be the motive of a virtuous action, but that the benevolent motive appears in this particular case to want its due degree of strength, and to be altogether unsuitable to its object. (TMS 304) 
Higher and lower motives were to be united in self-interest, but self-interest did not mean what the Chicago school of economics commonly supposes. If Smith had really said (which he did not) that self-interest was the foundation of society, it would have meant merely that society would properly be based on self-love and the virtues. What Smith actually said was much more explicit; the foundations of society was justice, 'Justice is the main pillar that upholds the whole edifice', because the whole object of the laws, without which society could not exist, was to constrain the useful motives of greed and self-love. The foundation of Smith's society was legal constraint on excessive self-love, and not self-love itself. 
The laws of justice would facilitate the division of labour, but economic growth did not depend entirely on the division of labour or the consequent extent of the market. It was limited primarily by the accumulation of capital, without which the division of labour could not proceed:
As the accumulation of stock must, in the nature of things, be previous to the division of labour, so labour can be more and more subdivided in proportion only as stock is previously more and more accumulated. (WN 277) 
But even the acquisition of inferior virtues required the conscious cultivation of impartiality to oneself:
In the steadiness of his industry and frugality, in his steadily sacrificing the ease and enjoyment of the present moment... the [inferior] prudent man is always supported and rewarded by the entire approbation of the impartial spectator.
Inferior prudence required self-command as well as self-love because, instead of providing for a different self in a future situation, the unconstrained ego would opt for immediate gratification:
The impartial spectator does not feel himself worn out by the present labour of those whose conduct he surveys; nor does he feel himself solicited by the importunate calls of their present appetites. To him their present, and what is likely to be their future situation, are very nearly the same. (TMS 215) 
[Impartial spectator = rationality / What if planning did not exist, only a different sort of gratification?]
Each action called for its own particular balance of virtue and self-love; the highest degree of virtue was required for the contemplation of moral philosophy and the formulation of the laws, but the same virtue was in a lesser degree required for commerce. Virtue could be expressed in commerce or in public life:
The superior wisdom of the good and knowing man directs others in the management of his affairs, and spurs them on to imitate his industry and activity. (JA 338) 
The laws of justice would encourage the extension of a complex web of trade and production, while the other virtues would help augment the capital stock and put the economy into rapid motion. 
But both the Lectures and the Wealth warned of the possible moral dangers of commercial life. As the division of labour extended, Smith noted, the variety of commodities and productive processes tended to increase, but the variety of work declined (WN 783). Although the ruling elite would find its life enriched by these innovations, the characters of most people were moulded by their mode of employment, and the enforced specialization of work would have a narrowing effect on the individual's psyche. The whole culture that went with the production of wealth would restrict the individual's intellectual horizons and dilute the spiritedness that war and public life required. 
As money values and the division of labour generated alienation within the capitalist system, the growing contradiction between the social and productive forces (commerce without virtue) would eventually produce a revolution that would transform society and the state (Smith's historical cycle). 
It was most important for the legislature to pass impartial laws, but the individuals who had to pass these laws were not impartial themselves. The demands of human nature and the 'rooted prejudices of the people' would make political impartiality almost impossible, and yet if the gulf between the impartial ideal and what was politically feasible were too great, society would break down. 'If the two principles coincide and act in the same direction, the game of human society will go on easily and harmoniously,' but otherwise 'society will be in the highest degree of disorder.' (TMS 234) 
Instead of conceding that 'the commercial system' was based on values, Smith presented its operation as the very opposite to a true value system. Amoral political systems confuse means with ends; and Smith presented the commercial system as a perverse and paradoxical system of power which really harmed the society whose interests it was supposed to promote. The commercial system aimed at low wages, when the only just and sensible aim of political economy was to encourage high wages, and its ultimate purpose was production, which was absurd because the only rational purpose of economic activity was consumption. The commercial system favoured the rich and powerful rather than the poor, and it made trade a source of international dissension when it should have been a cause for international co-operation.