I think it's easiest to think of Bitcoin TX fees as a bribe to the miners to include your transaction in a block sooner then other transactions that have a lower fee attached. You get to choose the fee that you provide to the miner.
The problem is that there are now so many transactions that are attempting to get through (And there is a maximum number of transactions that can be placed in a single block) that miners can be more selective. At this point your transaction is unlikely to get through without a minimum fee/bribe being provided because there are enough transactions with fees already at or above that level.
Stupid question.... What sort of compression is used on the block ? I.e If you use a better compression or re-align the transaction in the block can you not get more transactions per block ?
There is some compression at a storage level after blocks have been created, but it's not all that important. Pretty much all the data in Bitcoin is very difficult to compress with most of it being signatures, transactions, and addresses all stored in a binary format. The random nature of encryption makes it difficult to compress, and the formats have already been "hand tuned" for minimum size in many cases.
Not really, this is an intended feature of Bitcoin, not a bug.
Every so many blocks, the number of new Bitcoin handed to the miners is halved, and eventually it will drop to zero. When that happens, the only incentive for miners to continue mining (And thus keep Bitcoin going) is the fees collected from the transactions. So required fees are only going to go up.
The problem is that there are now so many transactions that are attempting to get through (And there is a maximum number of transactions that can be placed in a single block) that miners can be more selective. At this point your transaction is unlikely to get through without a minimum fee/bribe being provided because there are enough transactions with fees already at or above that level.