Most religions can be reconciled with sufficient esotericism. I personally practice something sort of like 'Buddhislam.' Let's start with the common ground and then go through a few problematic cases.
Buddhism's main philosophical goal is the cessation of suffering, which it does in two main ways. The 'sutric' method, the method practiced by mainstream Theravada and Mahayana Buddhists, involves relinquishing attachments through asceticism and gradual realization of the temporary and unsatisfactory nature of all experiences. This is usually accompanied by something called metta practice, or lovingkindness meditation, the idea being that you are replacing your discriminatory passions with something positive: pure unconditional love and compassion. The purpose is to reach equanimity essentially by starving the kleshas of opportunities to take root in oneself. The second method is the tantric: instead of denying the kleshas opportunity to arise, you embrace all aspects of experience without regard to any distinctions between good, bad, liked, disliked. The tantrika, by unconditionally accepting all experiences without aversion or discrimination, may thereby learn to see life as both the pure land and the charnel ground.
Islam does not have much to say about this theory one way or the other. The closest analogue would be tasawwuf (purification of the heart), from which we get the term 'Sufism'. I don't see anything incompatible here.
2. Self and the Ultimate
Central to all forms of Buddhism but much debated is the doctrine of anatta, or 'no self'. Usually this is taken to mean that what we call the self or ego is just a collection of constantly changing mental aggregates, influenced by experiences in our current and past lives, with no irreducibly basic component which we can identify as an eternal, unchanging 'I.' And yet paradoxically, Nirvana or Buddha-Nature is often described as being the 'true self'; as being luminous as well as empty.
There is actually a similar ambivalence about the nature of Ultimate Realization in the Sufi tradition, called fana. At times it seems the Sufis almost identify their 'true self' with God, but generally they are careful to qualify this as an annihilation of their own self in the divine essence. A quotation from Bāyazīd is illustrative:
“Then I became a bird, whose body was of Oneness and whose wings were of Everlastingness, and I continued to fly in the air of the Absolute, until I passed into the sphere of Purification, and gazed upon the field of Eternity and beheld there the tree of Oneness. When I looked, I myself was all those. I cried: ‘O Lord, with my egoism I cannot attain to thee and I cannot escape from my selfhood. What am I to do?’ God spoke: ‘O Abū Yazīd, thou must win release from thy ‘thouness’ by following my Beloved. Smear thine eyes with the dust of his feet and follow him continually.”
More information can be found here.
This identification of enlightenment with annihilation of the self in the Truth of Allah seems to offer a promising avenue for reconciling Islamic monotheism with Buddhism, viz. by an identification of Nirvana or Buddha-Nature with the God (but this identification also poses some difficulties, addressed below).
3. Afterlife and Rebirth
Traditional Buddhism says that, while there is no eternal unchanging 'self' (except perhaps in Buddha-Nature), nevertheless a chain of mental causation exists which could be crudely termed 'reincarnation'. This has sometimes been described as a stream of thought; prior mental habits and attachments exercising an influence on subsequent minds even beyond death, despite the lack of any material substrate to connect the previous 'life' with the next 'life.'
The Qur'an seems to give rather a different picture of the state of a person after death. Surah 23:99-103:
"Until, when death comes to one of them, he says, ‘My Lord, send me back,that I may do righteous deeds in the life that I have left.’ Never, it is but a word that he utters. And behind them is a barrier until the day when they shall be raised again. And when the trumpet is blown, there will be no ties of relationship between them that day, nor will they ask after one another. Then those whose good works are heavy — these will be prosperous; but those whose good works are light — these are they who ruin their souls; in Hell will they abide."
Since Buddhislam as a movement doesn't actually exist, at this point I am going to have to give my own understanding of these passages, which I believe comports with Buddhism.
First, it is clear that this passage is specifically about the wicked. Generally it is only middle-of-the-road people who are reborn back on the physical earth; the truly wicked incarnate in the infernal planes, while the good incarnate in heaven until their 'credit' runs out. The saint, or bodhisattva, or arhat, has achieved immortality and may choose to incarnate anywhere they desire.
The 'final judgment' of which the Qur'an speaks thus has a dual application: an individual who has reached immortality has been raised from the dead; a wicked person goes to the Fire, which either purifies them until they can again enter the physical world, or forcibly reintegrates their seed of consciousness with reality. The second application is to a historical spiritual age in which the residents of the infernal planes are prevented from incarnating on earth.
Basically, we all return to Allah, but we either do it the easy way or the hard way. Qur'an 101:8-9: "But as for him whose scales are light [who has no good deeds], Hell will be his [nursing] mother."
I died as a mineral and became a plant,
I died as plant and rose to animal,
I died as animal and I was Man.
Why should I fear? When was I less by dying?
Yet once more I shall die as Man, to soar
With angels blest; but even from angelhood
I must pass on: all except God doth perish.
When I have sacrificed my angel-soul,
I shall become what no mind e'er conceived.
Oh, let me not exist! for Non-existence
Proclaims in organ tones, 'To Him we shall return.'
4. Creator or No Creator?
The Qur'an strongly asserts that Allah is the creator of all things. This seems to directly contradict the Buddhist belief that there is no ultimate creator.
Buddhists often appeal to the Mulapariyaya Sutta in defense of this doctrine:
"He [the enlightened Buddha] directly knows Unbinding as Unbinding. Directly knowing Unbinding as Unbinding, he does not conceive things about Unbinding, does not conceive things in Unbinding, does not conceive things coming out of Unbinding, does not conceive Unbinding as 'mine,' does not delight in Unbinding. Why is that? Because he has known that delight is the root of suffering & stress, that from coming-into-being there is birth, and that for what has come into being there is aging & death."
The 'Unbinding' refers to Nibbana, the root of existence. At first glance this seems to flatly deny that the Ultimate Reality produces anything from itself, but I think we can get around this by deploying the Mahayana doctrine of two truths.
These two truths are the conventional and the ultimate. On the conventional level, we perceive reality as being made up of separate objects, which arise and pass away. To the conventional level belongs all of our everyday experiences. On the ultimate level, it is impossible to make distinctions between objects or to talk of arising and passing away.
I would argue that perceiving Allah as a separate 'creator' also belongs to the conventional. Sufi mystics often interpret Tawhid (the divine Unity) as meaning that Allah is the only existent, the only truth. The creation is thus to some extent illusory. It seems to have a real existence, but it doesn't. So when the Qur'an speaks about the creation, it (arguably) does so from the point of conventional reality, whereas the ultimate is more in line with Surah Al-Baqarah (2:115):
To Allah belong the East and the West; so withersoever you turn, there will be the face of Allah. Surely, Allah is Bountiful, All- Knowing.