Excellent question! The answer is that with Configuration Services, the load balancing/failover detection happens from the Config-Services generated client itself (hence point to point, and this provides some big advantages especially for intranet scenarios; albeit some disadvantages as well). Hence, to get load balancing ala the StockTrader web app to the middle tier stock trader business services (or from business service tier to Order Processor if running in that remote service config); this is predicated on the client app implementing Configuration Services. The Capacity Planner tool does not---so it will *not* automatically load balance requests against backend service tiers (Java or .NET nodes). I have though about re-implementing the Capacity Planner tool using Configuration Services (actually, not hard to do), such that its agents can auto-detect nodes of middle tier being tested, or user can use ConfigWeb to setup multiple backend servers to load balance against. This would also help in testing of failover characteristics of Config Services...but for now the Capacity Planner tool just works against a single node at a time.
There are of course other load balancing options; for example hardware or using Windows Network Load Balancing software built into Windows Server--that virtualize an IP address against mulitple hosts...if using this, then Capacity Planner can be pointed at the virtualized address, and load balance accordingly; but not a test of COnfig-service driven load balancing. I will try to add into Capacity Planner an implementation of Config Services; this might be handy for setting config options, etc as well as doing load balancing and failover ala StockTrader and other samples for Config Service that are part of the download.
(there are also a couple of bugs to address in the tool, I hope to do, plus I want to add binding/config options for testing with both transport and message-level security--to both .NET and Java implementations of the services).
1) Cannot verify any results I publish
2) You cannot tune a system to any degree
3) Cannot judge relative performance of various stacks; architecture tradeoffs within a given stack; different deployment options regarding location of front end web server, etc.
So for now it does what it does. One goal is that the two workloads chosen (much like a canned TPC or SPEC benchmark) are built using what I consider "standard building blocks" for the platforms they support; while simple, they do represent to some good degree the "raw" performance/capacity of software/hardware deployed for more sophisticated apps that will generally use the same core underlying building blocks. WSTest is about measuring just Web Service especially network stack/serialization/de-serialization performance (including .NET or any Java stack; and also .NET TCP-Binary encoding vs. HTTP-XML encoding--and important options WCF adds while simulatenously hosting HTTP-XML endpoints for interop). StockTrader is about introducing typical business logic behind the web service, involving database access (to SQL or Oracle); DB transactions including potentially distributed two-phase transactions against JTA or .NET Distributed Transaction Coordinator; etc. So both workloads are useful, even given current constraints of this pre-packaged tool.
If only I had time to add some level of scripting or auto-generation of test scripts based on WSDL/Meta-DataExchange of service operations...that would be really cool but all of a sudden I would be spending time building something like a poor-man's version of LoadRunner or even VS Load Test Tools; when these tools exist already (although free and turnkey has some advantages as well).
I really want to encourage people to do more load testing...I think not matter the platform, load testing early and often saves tons of money in the long wrong----finding out an option/feature/architecture decision does not scale early in the dev process could mean saving millions of dollars finding out after a complete system has been developed. Hence the crude but free Capacity Planning tool. And also I really want people to *believe* the results I publish; I do not just do this as an MS guy but also as a curious guy who wants to find the "truth" (which is out there, to parden an X-Files reference).
I do have plans for Capacity Planner, I will add to plans:
1) Implement Config Services to enable turnkey testing of Config-Service hosted apps with Config Services load balancing, node notifications etc as you note is lacking.....
2) Expand to include testing of WS-Security; both message level encryption performance and transport security; as well as mixed Transport with Message Credentials for a variety of options StockTrader already supports (does anyone really yet *know* the relative perf of IBM's stack (or other Java stacks) implementing message level security vs. .NET WCF; note there is at least one MSDN whitepaper on the perf costs of WCF message level security relative to transport security or "wide-open" SOAP on MSDN. But relative to other stacks? I for one want to know!
3) Might be interesting as well to add a Federated security scenario, at least with WCF, for finding about this given interest on this forum around STS.....
...performance. One big debate that I think will always rage is ORM-style apps/frameworks vs. more data-tier defined data access layers (more optimized, more flexible sometimes, less flexible other times). StockTrader sticks to the bread-and-butter: ADO.NET, largely using DataReaders; System.Transactions for transactions that promote automatically from single phase commit to two-phase commit/mid-tier tx logging--now on both Oracle and SQL Server backends); and use of a model class approach for passing data between tiers. Just like the WebSphere "Direct" mode of operations, that uses just a Stateless Session Bean to front-end direct access to DB from JDBC, vs. CMP and EJBs.
But interesting to test relative perf of both....my .NET style is akin to using JDBC/servlets in the Java world, and staying away from the ORM runtime stacks (although EJB 3.0 is better, that is a complete other architectural discussion (rant) I can go on, but should not right now)...I will note however that some of the more popular ORM open source technologies such as Hibernate are now also being implemented on .NET...sorry if I offended anyone...I am by no means the definitive expert....but have my opinions based on some level of experience; especially in the perf realm.
No comments:
Post a Comment