Magento and Ebay: 9 months later

March 4, 2012 12 Comments

It has been nearly 9 months ago since the Magento Developers Paradise 2011 and the announcement that Ebay has acquired 51% of the remaining Magento shares. You might think that 9 months is an unusual timeframe for a recap? Or that some kind of new idea or product has just been born? The answer is no! I’m writing this because I feel that I have to.

At first I’d like to sum up the positive events which happened during this period of time:

X.commerce Innovate Developer Conference 2011

I have attended the X.commerce Innovate Developer Conference 2011 in San Francisco and had a great time meeting new folks but also having a chat with people I haven’t seen for quite a while. I’ve also enjoyed some great and inspiring talks, especially the presentaion of Tom’s Shoes by Blake Mycoskie. Magento had it’s own track with sessions about Magento 2, Magento’s role in X.commerce, development, etc. During the keynote it became clear to me, that Magento will be one of the numerous parts of X.commerce. At this moment I was not sure if this will be positive or negative for the further development of Magento as an ecommerce platform.

The result of the event was that I’ve been planning to have a deeper look on the X.commerce fabric which was made public to download, but I haven’t done this until now!

Magento Developer Certification

A couple of weeks before the Innovate Conference Magento announced the beta phase of the Magento Developer Certification. This is something I was waiting for since 2009. It is a very important instrument to make the quality and development capacity of a solution partner transparent to the public (clients, developers seeking for a place to work with a good team, etc.). It was possible to take part at the exam right on the conference and I’ve received a voucher on site, so I gave it a shot and signed for the Developer Plus exam. The Developer Plus exam covers also the Enterprise Edition modules while the normal Developer Certification focuses on the Community Edition only. The exam was not easy, but working with Magento since the end of 2007 made it possible for me to pass without any preparation.

This accelerated my plan to set up an internal training program for my developers at the company I work for and also for a sibling company. Since last month I’m currently preparing 14 developers to take the Developer Plus exam!

Magento University

Offering training courses for developers is another crucial service to enable and support solution partners to deliver in time and in quality to their clients. I remember the days when you had to figure out everything by yourself by digging the code. And by everything I mean everything: even the architecture and how to write a proper module which doesn’t mess in the core and keeps the solution upgradeable. The lack of such services led to numerous failed projets or messed up solutions which didn’t satisfy the client. All of this is bad for the reputation of Magento.

With the Magento U Magento offers on site courses. The most important one is the course “Fundamentals of Magento Development” with a duration of 5 days. This is the perfect start for any developer who wants to work with Magento. I wish that there was a course like this in 2007!

In addition to the on site courses there are on demand video trainings which are a nice addition to my training program.

Magento Tests Automation Framework

This is the most recent publication: the Magento Tests Automation Framework. It is using PHPUnit to run about 1000 Selenium testcases to ensure if the core functionality of Magento is still working properly along with your customizations. Also it offers you to extend the tests to cover your customizations. Another thing I’ve been waiting for since a long period of time.

The lack of such tool leads to high testing effort: either you build such suite of testcases by yourself or you keep testing your projects manually. Both require high effort and under budget pressure this leads to poorly tested shops and again disappointed customers.

I didn’t have the time to tests the Magento Tests Automation Framework, but if the results will be good I will consider to implement it into our quality assurance process instead of our Selenium smoke test suites which we use at the moment.

Interlude

Sounds pretty good so far, doesn’t it? If there was not one more thing: Magento 2! Magento 2 is a very important project for Magento. It contains crucial architecture refactoring, decoupling of modules, performance enhancements, etc. The list is long and I’ve already written about it. Just releasing some new features, bigfixes and small improvements on indexing and stuff is not enough!

The release date was announced to be the end of 2012, so there is still some time left. My main concern is the lack of transparency. The last time I heard something about Magento 2 was in October 2011 at the Innovate Conference.

Lack of Transparency

Magento announced in June 2011 at the Magento Developers Paradise 2011, that the development of Magento will be performed using the Atlassian software stack which will also give the Magento community a better insight in the progress and development process.

So let’s see, what we’ve got:

I would say that there could be some more transparency than that! At least we still have GitHub: https://github.com/magento/magento2. Unfortunately, the development is not based on GitHub, there are only a couple of merges into GitHub. The last one was a couple of days ago, the commit before that was in the end of December 2011!

In fact, this is the reason why I’m writing this post. I’ve looked into the GitHub project about 4 or 5 days ago and until then the last commit was still from 2011 which concerned me a lot!

Conclusion

I think that Magento should meet its announcement to provide more transparency in the development process of Magento 2. I’m happy to read (after digging around for this information) that the public alpha will be completed in Q1 2012 and will contain the first version of the visual design editor, a modular admin application (so not everything is in Mage_Adminhtml !) and jQuery as the primary JavaScript library! It’s nice to announce that on the Magento Imagine, but you have to remember that the Magento community is not a bunch of salesmen which are happy to hear about the new stuff on conferences.

We are developers and like to be in touch with all the details! With every commit which might contain a fix for an issue reported by us! There are only 6 reported issues on GitHub, I think there could be way more of them if there would be more announcements and information feeds. Of course I know, that the majority of the community is working with Magento 1 and is very busy. But I think that Magento should enable it to deal with Magento 2 much earlier. That’s very important to enable a smooth migration to Magento 2. Which client will want to spend money to implement a store on Magento 1 in one year, when Magento 2 will hopefully be available as a stable release? You have to be prepared for that!

Magento should remember how the development process of Magento 1 was driven by the huge participation of the community which reported thousands of issues before Magento 1.0 was released. This is something which will be crucial for Magento 2 as well! For Magento and the community!

Advertisement

Filed under Solutions Tagged with , , , ,

Magento 2 Preview

June 19, 2011 20 Comments

A couple of days ago I wrote my first blog post about my experience at the Magento Developers Paradise 2011 on Ibiza and promised to dig deeper into the details of Magento 2, which I noted while listening to the talks by Yoav Kutner, Dmitry Soroka and Anton Makarenko. I think that this project is crucial for Magento to keep and develop its most precious asset: the Magento community. The Magento 2 project will show what the acquisition of Magento by Ebay really means for us. The release date is planned for mid/end of 2012.

Transparency

The first thing I want to mention is that Magento wants the development process be more transparent to the community. Therefore Magento will soon provide view access to the development tool stack, which consists of following Atlassian products:

  • Jira – issue and project tracking
  • Confluence – wiki for documentation
  • Bamboo – continuous integration
  • Crucible – code review tool
  • Fisheye – repository browser (thanks for the screenshot by Andreas von Studnitz in his blogpost about Magento 2)

Jira should give us a much better overview of the currently developed release and the planning of further releases. Confluence will provide a wiki with the Magento documentation. Bamboo should show metrics like broken builds ratio, unit tests count and code coverage percentage. Making Crucible public will provide a completely transparent code review process. I can’t really believe that this will be made public, because it is very intimate ;). And Fisheye will make browsing the code and looking on diffs between version smoother than now (actually you can only browse the directory tree in SVN and just look into files).

It was also interesting to hear that the Magento core team consist of 6 developers and is using Scrum.

Software Quality

Magento was and is often critisized that Magento lacks of automated tests and proper documentation. And in deed this leads to unstable releases and code which is hard to understand. At the moment maintaining Magento solutions is more expensive as it should be. The time to qualify a PHP developer to be able to work with Magento is quite long (approx. 6 months which is not only my opinion) and expensive. This makes it harder for Magento solution partners to scale their development capacity. In addition, the lack of automated tests makes it harder to develop and maintain customizations and even the core itself. Clients expect a stable solution and low maintenance costs from their Magento solution partner. Magento 2 is supposed to address these issues:

  • there are already approximately 1900 automated unit, integrity and performance tests for Magento 2
  • they already cover 40-50% of the code
  • the documentation process (inline and also external in Confluence) is implemented in the development process of Magento 2
  • there won’t be features which are released without automated test coverage and documentation
  • Magento tries to cut the time to qualify a developer by 50%
  • Magento considers porting the tests down to Magento 1 (I’m doubtful about this one)

This is nice to hear, but I recommend Magento to fulfil these announcement for Magento 2. I remember when I have visited the Magento office in Kiev for the first time in the mid of June 2009. I’ve discussed lots of topics with the team around Dima, Micha and Anton and we also talked about unit tests.

I'm the guy on the left, Anton, Micha and Dima from Varien from the right (2009)

A couple of weeks after the trip I saw this tweet: http://twitter.com/magento/status/2506514713 and was very excited about this news, but still we don’t have automated tests today.

The external documentation in Confluence is also a very good point. There is no official publication about the Magento architecture in form of a book. The first literature I’ve discovered was a book written by Mark Kimsal in 2008: php/architect’s Guide to E-Commerce Programming with Magento. It covers some basic architecture and digs into certain important modules. This was not really it, but it was better than nothing back then in 2008. In 2010 O’Reilly has published a German book called “Magento – Das Handbuch für Entwickler“, which I have co-authored. It went deeper into the core architecture and provided much more examples of how to apply the knowledge to practical tasks and issues. Unfortunately analyzing the book reviews today show, that people still have issues transferring the theory into real Magento development. Until now I haven’t heard of any other books which go deep into the core of the code. Most of the books concentrate on the installation, configuration and guiding admin users through the Magento Admin Panel functionality for the shop setup and the daily business of a Magento store after the Go Live.

Long story short: I’m very excited to see the official Magento documentation and hope that developers can benefit from it and that it can be used to educate Magento developers.

Frontend

I’ve enjoyed hearing that Magento 2 will throw out Prototype and extJs and will use jQuery only. My team at next commerce recently completed a project which target was to create a frontend theme which is using jQuery only and doesn’t contain any inline Javascript and CSS in the templates. It will be a component of our D2C software core and will be used in future D2C projects.

Magento 2 will have unlimited theme fallback: this means that you can build an unlimited theme inheritance tree which allows avoiding code redundancy in the templates, CSS and JS files completely. In Magento 1 there are only three levels of inheritance.

Example for multiple theme inheritance in Magento 2

A GUI design editor will provide functionality like moving blocks in layouts via drap&drop and it also will be able to export pages as one template file which will consist of all block templates and all states.The designer can edit that template instead of having to perform changes in lots of templates scattered in lots of different modules.

Architecture

Since Magento 2 is first of all a refactoring release, there will be a lot of changes to the architecture to improve scalability, extendability and quality of Magento.

In Magento 2 there won’t be any magic getters and setters. This means that the interfaces to all objects in Magento have to be strictly declared in the code. Also the data model definition will be declared, so that the database is no more the source of the data model definition of objects (this will actually have an impact on EAV). Together with the stricter documentation policy we should have finally a proper API documentation for Magento 2.

Currently, Magento objects like products, orders and customers are contaminated with things like controlling the persistance of their data (load, save, delete). There will be a strict separation of domain models (define which characteristics objects like products have) and service models (perform operations on objects).

If you install two Magento extensions which are rewriting the same model, block, helper, etc. from the core, one of the extensions won’t work, because the first extensions wins when it comes to rewriting. In Magento 1 the only solution for that is to modify one of the modules to build an inheritance chain, so that the modules both work again (of course this only works if the extensions change different characteristics of a class). Magento 2 will allow to configure inheritance chains of models from additional extensions without code changes, which makes it easier to upgrade these extensions. This is a feature which OXID eSales already has.

To be able to disable functionality easier, Magento 2 will have less coupling between modules. Magento 2 introduces components, which consist of tightly coupled modules: Sales (Sales, Checkout), Catalog (Catalog, CatalogIndex, CatalogSearch, …), CMS. These components can be disabled without breaking the rest of the system, so it will be possible to disable Sales to have just a browseable catalog and static content (CMS) or just only static content on the site, if you disable Sales and Catalog.

Modules in Magento 2 will contain all files which belong to the module in one folder: app/code/<pool>/<space>/<model>. Not only PHP classes, but also the templates and skin files will be in this path. The filesystem structure will be rearranged: there will be a folder called “pub”, which will contain all files which should be accessible to the public (jslib, media, skins). The other directories (app, dev, lib, var) should not be delivered by the webserver directly. See a screenshot in Alexander von Studnitz post about Magento 2.

Objects (models, helpers, blocks, etc.) will be instantiated using this pattern: createObject(‘<class name>’) (e.g. createObject(‘Mage_Catalog_Model_Product’)) and not with Mage::getModel(‘catalog/product’). The method is of course still a factory which looks into the configuration for rewrites. Also the getTable-method will be changed: you will have to use getTable(‘catalog_product_entity’) instead of using the alias getTable(‘catalog/product’). The method will return the table name which might have a table prefix.

These changes have an impact on the compatibility of Magento 2 to Magento 1: Magento 1 extensions won’t work with Magento 2. It will be required to migrate extensions to be compatible with Magento 2. Magento will provide a documented migration path for that. I hope there will be a database migration tool, too.

Persistence

Magento 2 will provide support for the RDBMS MySQL, Oracle and MS-SQL.The refactoring of the resource model is already announced for the upcoming CE1.6 and EE1.11 release and will make following changes to the architecture of the persistence layer of Magento:

Refactoring of the resource model layer in Magento 1

The advantage of the refactoring is that the non-RDBMS-specific code is in the Resource Model and everything which is RDBMS-specific is removed and put into a helper for each supported RDBMS. This makes it easier to introduce support for additional RDBMS with nearly no code redundancy. The helpers are only needed when it is required to go beyond the limits of the database abstraction layer provided by Zend_Db for better performance or usage of additional functionality which other RDBMS do not support. Also the install and upgrade scripts will be abstract (upgrade-0.0.1-0.0.2 instead of mysql4-upgrade-0.0.1-0.0.2) and only when it is required, Magento 2 will still support mysql-*, oracle-*, etc. scripts for RDBMS-specific code. This refactoring will be backwards compatible.

A breaking news for me was that Magento is considering to remove EAV and only provide the EAV database schema for the catalog as an optional mode. This would improve the performance significantly, because there will be less JOINs in the queries and therefore less temporary tables which have to be created to sort and filter such results in MySQL. Also Magento would require less indexing which makes data imports and modifications of data faster.

EAV was build in since the start to support the modification of data models for products, categories, orders, quotes, customers and customer addresses without changes to the database schema. So basically it was possible to change the data model of these entities during the runtime of Magento in the Admin Panel. I was involved in a couple of medium- to big-size ecommerce projects on Magento in my career and I can say, that such kind of functionality is not needed and it also can be very dangerous for the stability of the Magento solution. Changes to the data model have to be part of a release and have to go through a proper release process which consists of QA activities such as testing the changes on a staging environment. This requires to perform such kind of changes by using upgrade scripts to ensure that the data model of all entities is the same on all systems (dev, acceptance, staging, production). And since the changes are part of a release and have to be deployed it is not a big deal that there are changes to the database schema.

Everybody who has watched the evolution of Magento 1 has also noticed, that EAV was removed step by step from Magento to gain better performance: first the quotes were stored in flat tables instead of the EAV schema, then the orders followed. Order items were flat right from the start (if I remember correctly) to be able to generate sales reports without too much pain. The flat catalog was introduced to remove the overhead which slowed down the rendering of category and product pages. The flat catalog is a special kind of an index, where each store has its own category and product table.

For me the logical next step is to remove the EAV for customer and customer addresses and last but not least from the catalog.

Web API

Although Magento has often announced to extend the API to cover more functionality, the API is still way behind the possibilities you have in the Admin Panel. Magento 2 will address this issue by including the extension of the API into the development process of new functionality. So all new features will be released together with new API calls covering the new functionality. Also the API will support JSON-RPC and REST in addition to SOAP and XML-RPC which are supported in Magento 1.

Maintenance

There are a couple of things which should improve the maintenance of running Magento sites.

The upgrade process will have a better workflow: the upgrade will be performed on a staging copy of the store (code and database). This staging environment can be used for tests before deploying the upgrade on the production environment. Magento solution partners should all do this already now with own deployment scripts or manually on staging environments. If not: shame on them!

The Magento profiler will output profiling entries in a hierarchy and not as a plain list like in Magento 1. This makes the profiler output easier to read. Also the output format can be configured to support Firebug and other browser plugins.

Wrap Up

At first I was expecting Magento 2 to go with the latest trends in the world of web applications. So called post-modern web applications (e.g. GMail) leave the classical request-response principle for loading complete HTML pages. They are designed like a native client-server application which has a powerful view running on the client and just fetches raw data from the server. I watched a nice presentation on the next09 by Malte Ubl in 2009, who was a Technical Director at SinnerSchrader at that time and now has joined Google in the end of 2010, with the title “Building post-modern Web Applications”. It covers this topic and you can watch it here: http://video.nextconf.eu/video/923175/building-post-modern-web.

Today, I think that the approach of Magento to make a major refactoring release is more pragmatic than changing the architecture of Magento completely. It is possible to release earlier and remove a lot of urgent issues outlined above faster to help the community to deliver better solutions for clients, better extensions for the community and better feedback for Magento. The benefits of this will form a better foundation for a big step to a modern web application design. The big question is how the X. Commerce strategy will influence the evolution and especially the revolution path of Magento. Will Magento be able to perform such big steps being a part of X. Commerce?

Filed under Solutions Tagged with , , , , , , , , , , , , ,

Magento Developers Paradise 2011 Wrap Up

June 12, 2011 5 Comments

Last week on the 4th of June 2011 I’ve started my trip from Hannover to Ibiza to attend the annual Magento Developers Paradise(MDP). I was already looking forward to the event because I have enjoyed my time at the Magento Developer Paradise on Mallorca in October last year. The atmosphere at this kind of locations is very nice and relaxed, everybody is in a good mood and enjoys the sun and free drinks during the breaks and after the talks.

Unfortunately my plane was late, so that I’ve missed my connection to Ibiza, had to spend some hours on the airport in Palma de Mallorca and missed the conference kickoff dinner. But on a MDP this is not a problem: even if you join at 11pm you still meet most of the attendees at the pool bar. So did I and had a chat with Dmitriy Soroka, lead system architect at Magento Inc., Thomas Fleck, CEO of Netresearch and host of the event, and lots of other people. Until 2am I’ve been chatting about Magento topics and what I expect of the upcoming Magento 2.

The next day started with having a breakfast together with Yoav Kutner, CTO of Magento Inc., and some other Magento core developers. After this at 9am Yoav held his keynote about the current status of Magento and what topics will be covered in the upcoming talks. The most interesting was of course Magento 2 which will be released mid/end of 2012. The development has already started, but not all decisions are final. The release will be a major refactoring release which will focus on software quality and performance. To achieve this, a lot of changes to the architecture will be performed, so that Magento 2 won’t be backwards compatible with Magento 1 extensions.

Yoav Kutner is helding the keynote for MDP11

Right after the keynote Andrey Tserkus from Magento Inc. held his presentation “Breaking news: Magento starts running on multiple RDBMS”. This feature will already be available in the releases Magento EE1.11 and CE1.6 next month. Magento will focus on the support of Oracle and MSSQL first, but will add more RDBMS in the future. This feature contains the refactoring of the resource model of the Magento core: in current releases there are only “Mysql4” resource models which contain MySQL specific code but also abstract database access by using Zend_Db functionality which allows using multiple RDBMS. In EE1.11/CE1.6 the resource models will be cleaned up and will only contain non-RDBMS-specific code. The RDBMS-specific code will be moved into resource model helpers. This enables to add support for other RDBMS with less effort since most of the code is in the abstract resource model. Only when it is required to leave the DB abstraction layer e.g. for achieving better performance you need to write code in the resource model helper for the specific RDBMS. The refactoring of the resource model is backwards compatible, so that custom Magento modules still will work using MySQL.

I have also enjoyed the following talks about checkout customization by Vitaliy Korotun (Magento Inc.) and extending Magento hosting capabilities by Philippe Humeau (nbs System). In the afternoon everybody was invited to join a boat trip to Formentera, a small and beautiful island next to Ibiza. We were on board of a catamaran so it was possible to go swimming and we had a delicious dinner on it, too. In the evening we were brought back to our hotel and you could meet plenty of attendees at the pool bar afterwards. Some of them went out to hit the bars and the clubs at night. The nightlife on Ibiza is amazing!

Catamaran trip

The next day had very exciting talks about Magento 2 by Dmitriy Soroka, lead system architect at Magento Inc., and “MVC – model lessons learned” by Anton Makarenko, member of the Magento core team at Magento Inc.. I will only sum up the, in my opinion, most important news on these topics. I will dig deeper into them and write down my thoughts in a follow up post about Magento 2.

  • transparent development process using the atlassian software stack and making that available for the community (continious integration, documentation, issue tracking)
  • automated unit, integrity and performance tests are available for Magento 2; there are already 1900 tests
  • the documentation will be included in the development process, so it is always complete and up to date
  • jQuery will be used in the frontend, no Prototype and no extJs
  • unlimited theme fallback, you can create unlimited theme inheritance trees
  • Magento considers the removal of EAV and only offer it as an optional mode for the catalog
  • domain structural integrity: there will be a strict separation between domain models and service models
  • removal of magic getters and setters enforcing strict data model definition and interface declaration for models
  • configurable inheritance chains to be able to resolve incompatibilities between modules; OXID eSales has this feature already
  • the Core API will be extended in parallel to the development of new features, so it will cover all functions; it will support SOAP, XML-RPC, JSON-RPC and REST
  • less coupling between modules: Magento will introduce components, which consist of modules which are tightly coupled; these components (e.g. Sales, Catalog) will be loosly coupled to the rest of Magento and can be disabled

I didn’t mention lots of things and also was not able to write everything down during the talks, but I think this gives a good overview of all improvements which are planned for Magento 2. If Magento will be successful in realizing them, this will be a huge jump forward in software quality and transparency and this should distinguish Magento from other open source shop platforms.

At 8pm everybody gathered in the lobby for the night event. We went by bus to the historic city of Ibiza and had a guided tour to the fortification and the historic city of Ibiza, called Dalt Vila. After the tour we went to the famous Zoo Bar and had some drinks before getting back to the hotel. Like on the previous day, a large group of people bought club tickets and had an amazing night in the Privilege club.

Great view from the fortification in the historic town of Ibiza

On the last day you could chat to people sitting at the pool bar and spending some time before taking a cab to the airport. You could hear most of them discussing the news about the acquisition of Magento by Ebay and that Magento will be at the core of the new X.Commerce platform. Some people were sceptical, some were optimistic. I’m excited of how this will influence the Magento products and the community.

In the late afternoon I finally took a cab together with the team of Netresearch (we had the same flight from Ibiza to Mallorca) and had a good trip back home.

To sum it up: it was an amazing event with lots of good talks, interesting people and a very nice and relaxed atmosphere for networking. I can recommend this event and already plan to attend next year! I also wanted to thank Alice Patzer, Andrea Panajotowa, Annemarie Tappert and Thomas Fleck for organizing this event and Rico Neitzel for shooting lots of nice photos. You can see them here: http://www.flickr.com/photos/34764379@N04/sets/72157626906928668/

Filed under Solutions Tagged with , , , , , , , , ,