TACC-Intel Symposium Highlights MIC Architecture Developments

By Aaron Dubrow

May 1, 2012

Leading researchers, hardware and software engineers, and high performance computing specialists from around the country attended the TACC-Intel Highly Parallel Computing Symposium on April 10 and 11 at the Texas Advanced Computing Center (TACC) in Austin, Texas. The meeting showcased the experiences of researchers who had ported their scientific computing codes to Intel’s Knights Ferry (KNF) software development platform, — the prototype hardware and software development package for Intel’s Many Integrated Core (MIC) architecture — as well as those working on the single-chip cloud (SCC). 

Over 100 participants from all sectors of the science and technology community attended the symposium and more than 30 stayed on to attend a tutorial on “Preparing for Many Core Programming,” taught by Kent Milfeld, research associate at TACC.  By and large, participants expressed enthusiasm for the forthcoming chips based on their early experiences. The symposium followed a series of Intel-led discussions and training sessions held around the world over the last six months in anticipation of the release of Knights Corner, the first product to use the Intel MIC architecture.

Day One

Two long-time Intel engineers, James Reinders and Tim Mattson, anchored day one of the symposium. Both researchers had worked on ASCI Red, the first supercomputer to break a teraflop of sustained performance in 1996. In addition, Reinders had worked on Knights Corner, the first commercial supercomputing chip to achieve one teraflop, which debuted at the Supercomputing 2011 conference.

The invited speakers steered clear of discussions about the raw performance or architectural details of the Intel MIC chip itself, the ostensible topic of the workshop, but still being closely held by Intel. Instead, they focused on the goals and research processes that led to the development of Intel’s MIC architecture, and the ecosystem of libraries, kernels and programming paradigms that Intel hopes will make its new coprocessors a long-term success in the HPC community.

In his talk, Reinders described the suite of tools and libraries that Intel has developed in anticipation of their manycore chips, as well as their efforts to improve the open standards that enable parallel programming. He expressed the oft-repeated goal, voiced by many Intel staffers, of making the Intel MIC coprocessor usable to researchers right out of the gate. 

Reinders spoke at length about the toolkit developed by Intel called Threading Building Blocks, which assists in the implementation of threading processes, while providing capabilities for task scheduling, synchronization and memory allocation. A thread is the smallest unit of processing that can be scheduled by an operating system. On a multicore system, many threads can run at the same time with each core running a particular thread or task.

“If your program can scale, Intel Threading Building Blocks won’t get in your way,” Reinders said. “There’s little penalty to using it.”

He also noted Intel’s acquisition and subsequent development of Intel Cilk Plus, a sister to Intel Thread Building Blocks for the C and C++ languages. According to Reinders, Intel Cilk Plus offers an easy, reliable way to improve the performance of programs on multicore processors.

Vectorization was a buzzword throughout the symposium. In parallel computing parlance it refers to software programs that are modified to perform the same operation many times simultaneously on a large number of operands. Vectorizing a code allows it to achieve significantly better performance, as much as four times as many operations per clock period on a CPU, and even more on a manycore chip like the future Knights Corner. Reinders was the first, but not the last of the Intel speakers to highlight the importance and value of vectorizing and threading one’s code.

Intel researcher, Tim Mattson, the second invited speaker, works on parallel system design, parallel programming environments and application software that are sometimes “far off the roadmap.” He pushed the date of the first teraflop chip back a few years to his time working on Intel’s 80-core Teraflops Research Chip, a “flop monster” on which the company tested its manycore ideas and architectures.

More than anything related to Intel MIC architecture, Mattson’s talk gave a compelling picture of the way Intel operates, and the concern, care and angst shared by its engineers as they create a new product line for the multi-billion dollar company. According to Mattson, the power problem looms largest in his group’s research, overshadowing even performance gains.

To minimize energy consumption, Mattson and his colleagues conducted a component-by-component power analysis of the experimental chips on which they worked. On the 80-core chip, they found that 28 percent of the power was going into networking capabilities. After tweaking the design, energy consumption was down to 10 percent on the 48-core, version. These two development chips, along with Larrabee, shaped the theoretical basis for Intel’s MIC architecture, and researchers from the three projects comprise the core Intel MIC team.

More than a dozen research groups at Intel are working on the manycore problem from a variety of angles. How many cores can you fit on die? What architecture should many-core chips have? Can such chips be manufactured at reasonable cost? How do you implement a clock infrastructure that can scale?  “It’s the small things that will kill you,” Mattson said.

He also spoke of future-looking ideas such as fine-grained management of the voltage, frequency and power consumption of compute cores that can be controlled by the programmer. Many challenges exist with such a solution, but Mattson’s description of the project provided a glimpse of the kinds of ideas Intel is exploring in their experimental research labs. “The architecture for many-core is still being determined,” he said.

Along the road to debugging and developing software for its manycore offerings, Intel went in an unusual direction. Instead of hiring a large internal software development group, Intel partnered with hundreds of institutions worldwide — among them TACC, many U.S. national laboratories, and top research institutions in the U.S., Europe and Asia — to test-drive the prototype MIC equipment.

“We built the silicon and said, ‘Let’s look to the academic community to do the research on it,’” Mattson explained in his talk.

Since late 2010, this elite group, the Many-Core Application Research Community (MARC), has been experimenting with the Single-Chip Cloud Computer (SCC) experimental processor, a 48-core ‘concept vehicle’ created by Intel Labs as a platform for manycore software research. More recently, researchers have been experimenting with the Knights Ferry hardware as well. Through these initiatives, researchers have implemented solutions that Intel never would have thought of, according to Mattson. “The smartest people in the world don’t necessarily work at your company,” he admitted.

This sentiment served as an apt segue to the paper and poster sections of the symposium, where early users presented results from code porting, testing and performance characterization.

The two-day event pointed to the promise and challenge of implementing existing codes on Intel’s new coprocessor. No absolute performance results were offered, yet scalability plots, comparative studies, and anecdotal experiences provided an overall positive impression of the group’s experiences to date. Most impressively, several groups reported independently that their applications ran right away on Knights Ferry with no or very minor code modifications.

Symposium papers and presentations are available online. A sampling of presentation briefs follows:

Robert Harkness: “Experiences with ENZO on the Intel MIC Co-Processor”

Harness (National Institute for Computational Sciences/Oak Ridge National Laboratory) is one of the creators of ENZO, a large cosmological community code used to study the early ages of the universe. Harkness estimated that it had taken 25 person-years to develop ENZO. The code can’t be abandoned, he said, but no one will pay to rewrite it either. The question he posed was: how to move forward with a large, working science code?

Harkness tested the application on three KNF systems at NICS, which cover most of the choices in basic cluster configurations:

  • An Intel development workstation with two Intel  Xeon  5600 processors and 2 KNF cards.
  • A Cray CX-1 system consisting of a single head node with two Intel Xeon 5600 processors
  • And two compute nodes each containing two Intel Xeon 5600 processors plus one KNF card.
  • An Appro cluster consisting of four compute nodes, each of which contains two Intel Xeon 5600 processors and two KNF cards, for a total of eight KNF cards

He tested the Knights Ferry in both native and offload modes and presented relative scaling results for both.

Harkness said he was personally able to port his code to KNF in one week. In his paper he wrote: “With all these components in place for offload and native builds, the compilation of ENZO-R itself is straightforward. No ENZO-R source code modifications are required to build a functional ENZO-R binary in native mode although, clearly, some source code changes will be necessary to achieve full optimization on the Intel MIC co-processor.”

After the experience, Harkness said he is “bullish on Intel MIC co-processors.” However, he warned the chip community to not waste time on vendor-specific interfaces.

Karl W. Schulz: “Early Experiences Porting Scientific Applications to the Many Integrated Core (MIC) Platform”

Schulz is the director of the scientific applications group at TACC and also chief software architect for The Center for Predictive Engineering and COmputational Sciences (PECOS), a DOE-funded Center of Excellence at The University of Texas at Austin. The group was funded four years ago to develop new uncertainty quantification (UQ) methodologies and apply them to multiscale simulations of atmospheric reentry vehicles.

He and his collaborators tried several codes from their portfolio on Maverick, a Knights Ferry test system with eight nodes and one Knights Ferry per Westmere node, and also built 3rd party libraries to facilitate runs. He presented relative scaling results and anecdotal experiences from several application tests.

Included among these applications was the porting of a quantum chemistry application that had 1.4 million lines of Fortran code.  They also tested the kernel of a Direct Numerical Simulations (DNS) code for turbulent fluid flow, a multi-threaded FFT kernel that exhibited good strong-scaling properties on the Knights Ferry. In the future, the group is targeting very large turbulent flow simulations (10 billion grid points) and proper utilization of the Intel MIC coprocessor will be crucial for achieving that goal.

Schulz who won best paper at the symposium, gave the most comprehensive insights into the process that MARC members have been experiencing over the past few months. To the assembled application developers in the room he counseled that “moving forward, we must take vectorization more seriously.” Overall, he remarked, “It was a positive experience for all of us.”

Ashay Rane: “PerfExpert and MACPO: Which code segments should (not) be ported to MIC?”

Rane is one of the developers of the tools PerfExpert and MACPO, which are designed to analyze the performance and performance characteristics of multi-threaded applications on multicore chips. In his spirited talk he presented a process by which application developers can reliably determine the parts of a code that are good candidates for execution on Intel MIC co-processors. Guidance in this area is certainly needed. Focusing on suitable candidates for porting (either whole functions or loops nests) has the potential to not only greatly reduce the level of human effort, but also to increase the achievable speed-up.

In Rane’s two-step approach, PerfExpert is used to identify the hot spots in an application. MACPO, currently under heavy development by Rane and others, analyzes data access patterns and TLB misses within the hot spots to eliminate those that exhibit irregular access strides, etc. Interestingly, the analysis is not done on the Intel MIC co-processor itself, but on the host, which enables users today, who do not have access to any Intel MIC platform yet, to prepare using this tool.

Three examples were presented as a proof of concept. Those code regions that were recommended by the tools did indeed perform best on the Knights Ferry Software Development Package (SDP).

Sreeram Potluri: “Intra-MIC MPI Communication using MVAPICH2: Early Experience

Sreeram Potluri is a Ph.D. student at The Ohio State University who has been researching and developing the MVAPICH2 library for MPI on InfiniBand and other communications platforms. Potluri describe his group’s work implementing MVAPICH2 for communications between cores on the Knights Ferry platform.

Their initial experiments ported MVAPICH2 directly to the Intel MIC card with no changes. Subsequently, they developed two kinds of optimizations of the library. The first tuned its parameters to take maximal advantage of the memory hierarchy and cache structure on the Knights Ferry card. The second set of optimization extended the work done in the first to use Intel MIC-specific communications protocols to further improve the performance.

The results showed impressive performance gains relative to the baseline case for all varieties of communications operations that Potluri tested. These included direct, point-to-point communications between pairs of cores, and collective communications operations that gather information from many cores together or spread information from one core to all the others.

Future work will take advantage of protocols for communication between Intel MIC cards in the same host, between tasks on the Intel MIC card and tasks on the hosts, and between Intel MIC cards running in different host computers. Potluri received the award for best student paper at the symposium.

Day Two

The day two invited talks focused more explicitly on the Intel MIC coprocessors, the programming models that will drive usage, and the market conditions that inspired Intel to develop the coprocessor.

Scott McMillan, a staff software engineer at Intel, spoke on “Programming Models for Intel Xeon processors and Intel MIC Architecture.” His presentation highlighted an important point: that Xeon and Intel MIC co-processor will work together in most installations.

TACC’s upcoming Stampede system, which will be the first large-scale deployment of the Intel MIC co-processors, has a two-petaflop base system built on the Intel Xeon E5 2600 processors (formerly codenamed Sandy Bridge). The Knights Corner coprocessors will add eight petaflops of expected peak performance. Many of the discussions about parallelism, threading and vectorization apply equally to Intel’s multicore Xeon processor and to the Intel MIC architecture.

The combination of these two architectures lend themselves to a spectrum of programming models and mindsets, which McMillan outlined in his talk, from all multicore, to offload, to all manycore, to symmetric usage of both the Intel Xeon and the Intel MIC architectures.“Which to use depends on the shape and structure of one’s problem,” he said.

The keys to productive performance on the Intel MIC architecture are a matter of common sense, McMillan insisted, and could begin immediately. He offered a few expert tips:

  • Pick the right model for your application.
  • Vectorize your application (today).
  • Parallelize your application (today).
  • Go asynchronous to overlap computation and communication

“Hybrid programming today prepares you for Intel MIC architecture,” he concluded.

Last up was Nash Palaniswamy from Intel’s Technical Computing Group, who offered some perspective on the strategy behind Intel’s manycore architecture.  For more than 15 years, Palaniswamy has been working in the accelerator and coprocessor space. Throughout that time, technologically-driven companies in the oil and gas, financial services and pharmaceutical industries, as well as academic and government researchers, have explored the use of accelerators to find answers quickly, driving strong research in this market.

Intel’s development of manycore chips over several years shows a unique level of commitment to enhancing performance, Palaniswamy said. In 2006, Intel began its accelerator and coprocessor effort under the Intel QuickAssist Technology program, and in 2010 after a thorough analysis of 80 to 90 workloads from customers, they announced a roadmap for manycore. According to Palaniswamy’s assessment, manycore processors are useful for 10 to15 percent of applications today, and this number will continue to grow as programmers increasingly think parallel.

In the road-map, “little cores” — smaller, low-power cores with reduced single thread performance that rely on a high degree of parallelism — co-exist and supplement “big cores” – higher-power, higher-performance cores. This change, Palaniswamy insisted, could be achieved in a way that maximizes code reuse while requiring minimal developer re-training.

“We want to make sure the software that runs today will run 10 years from now with minimal changes,” Palaniswamy said. “Architecture, models and applications evolve. Revolution is unnecessary and unaffordable.”

With scientists continuing to port, test and optimize their software for Knights Ferry, and ultimately Knights Corner, conversations about the Intel MIC architecture will be ongoing for many months to come.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

AWS Embraces FPGAs, ‘Elastic’ GPUs

December 2, 2016

A new instance type rolled out this week by Amazon Web Services is based on customizable field programmable gate arrays that promise to strike a balance between performance and cost as emerging workloads create requirements often unmet by general-purpose processors. Read more…

By George Leopold

AWS Launches Massive 100 Petabyte ‘Sneakernet’

December 1, 2016

Amazon Web Services now offers a way to move data into its cloud by the truckload. Read more…

By Tiffany Trader

Weekly Twitter Roundup (Dec. 1, 2016)

December 1, 2016

Here at HPCwire, we aim to keep the HPC community apprised of the most relevant and interesting news items that get tweeted throughout the week. Read more…

By Thomas Ayres

HPC Career Notes (Dec. 2016)

December 1, 2016

In this monthly feature, we’ll keep you up-to-date on the latest career developments for individuals in the high performance computing community. Read more…

By Thomas Ayres

Lighting up Aurora: Behind the Scenes at the Creation of the DOE’s Upcoming 200 Petaflops Supercomputer

December 1, 2016

In April 2015, U.S. Department of Energy Undersecretary Franklin Orr announced that Intel would be the prime contractor for Aurora: Read more…

By Jan Rowell

IBM and NSF Computing Pioneer Erich Bloch Dies at 91

November 30, 2016

Erich Bloch, a computational pioneer whose competitive zeal and commercial bent helped transform the National Science Foundation while he was its director, died last Friday at age 91. Bloch was a productive force to be reckoned. During his long stint at IBM prior to joining NSF Bloch spearheaded development of the “Stretch” supercomputer and IBM’s phenomenally successful System/360. Read more…

By John Russell

Pioneering Programmers Awarded Presidential Medal of Freedom

November 30, 2016

In an awards ceremony on November 22, President Barack Obama recognized 21 recipients with the Presidential Medal of Freedom, the Nation’s highest civilian honor. Read more…

By Tiffany Trader

Seagate-led SAGE Project Delivers Update on Exascale Goals

November 29, 2016

Roughly a year and a half after its launch, the SAGE exascale storage project led by Seagate has delivered a substantive interim report – Data Storage for Extreme Scale. Read more…

By John Russell

AWS Launches Massive 100 Petabyte ‘Sneakernet’

December 1, 2016

Amazon Web Services now offers a way to move data into its cloud by the truckload. Read more…

By Tiffany Trader

Lighting up Aurora: Behind the Scenes at the Creation of the DOE’s Upcoming 200 Petaflops Supercomputer

December 1, 2016

In April 2015, U.S. Department of Energy Undersecretary Franklin Orr announced that Intel would be the prime contractor for Aurora: Read more…

By Jan Rowell

Seagate-led SAGE Project Delivers Update on Exascale Goals

November 29, 2016

Roughly a year and a half after its launch, the SAGE exascale storage project led by Seagate has delivered a substantive interim report – Data Storage for Extreme Scale. Read more…

By John Russell

Nvidia Sees Bright Future for AI Supercomputing

November 23, 2016

Graphics chipmaker Nvidia made a strong showing at SC16 in Salt Lake City last week. Read more…

By Tiffany Trader

HPE-SGI to Tackle Exascale and Enterprise Targets

November 22, 2016

At first blush, and maybe second blush too, Hewlett Packard Enterprise’s (HPE) purchase of SGI seems like an unambiguous win-win. SGI’s advanced shared memory technology, its popular UV product line (Hanna), deep vertical market expertise, and services-led go-to-market capability all give HPE a leg up in its drive to remake itself. Bear in mind HPE came into existence just a year ago with the split of Hewlett-Packard. The computer landscape, including HPC, is shifting with still unclear consequences. One wonders who’s next on the deal block following Dell’s recent merger with EMC. Read more…

By John Russell

Intel Details AI Hardware Strategy for Post-GPU Age

November 21, 2016

Last week at SC16, Intel revealed its product roadmap for embedding its processors with key capabilities and attributes needed to take artificial intelligence (AI) to the next level. Read more…

By Alex Woodie

SC Says Farewell to Salt Lake City, See You in Denver

November 18, 2016

After an intense four-day flurry of activity (and a cold snap that brought some actual snow flurries), the SC16 show floor closed yesterday (Thursday) and the always-extensive technical program wound down today. Read more…

By Tiffany Trader

D-Wave SC16 Update: What’s Bo Ewald Saying These Days

November 18, 2016

Tucked in a back section of the SC16 exhibit hall, quantum computing pioneer D-Wave has been talking up its new 2000-qubit processor announced in September. Forget for a moment the criticism sometimes aimed at D-Wave. This small Canadian company has sold several machines including, for example, ones to Lockheed and NASA, and has worked with Google on mapping machine learning problems to quantum computing. In July Los Alamos National Laboratory took possession of a 1000-quibit D-Wave 2X system that LANL ordered a year ago around the time of SC15. Read more…

By John Russell

Why 2016 Is the Most Important Year in HPC in Over Two Decades

August 23, 2016

In 1994, two NASA employees connected 16 commodity workstations together using a standard Ethernet LAN and installed open-source message passing software that allowed their number-crunching scientific application to run on the whole “cluster” of machines as if it were a single entity. Read more…

By Vincent Natoli, Stone Ridge Technology

IBM Advances Against x86 with Power9

August 30, 2016

After offering OpenPower Summit attendees a limited preview in April, IBM is unveiling further details of its next-gen CPU, Power9, which the tech mainstay is counting on to regain market share ceded to rival Intel. Read more…

By Tiffany Trader

AWS Beats Azure to K80 General Availability

September 30, 2016

Amazon Web Services has seeded its cloud with Nvidia Tesla K80 GPUs to meet the growing demand for accelerated computing across an increasingly-diverse range of workloads. The P2 instance family is a welcome addition for compute- and data-focused users who were growing frustrated with the performance limitations of Amazon's G2 instances, which are backed by three-year-old Nvidia GRID K520 graphics cards. Read more…

By Tiffany Trader

Think Fast – Is Neuromorphic Computing Set to Leap Forward?

August 15, 2016

Steadily advancing neuromorphic computing technology has created high expectations for this fundamentally different approach to computing. Read more…

By John Russell

The Exascale Computing Project Awards $39.8M to 22 Projects

September 7, 2016

The Department of Energy’s Exascale Computing Project (ECP) hit an important milestone today with the announcement of its first round of funding, moving the nation closer to its goal of reaching capable exascale computing by 2023. Read more…

By Tiffany Trader

HPE Gobbles SGI for Larger Slice of $11B HPC Pie

August 11, 2016

Hewlett Packard Enterprise (HPE) announced today that it will acquire rival HPC server maker SGI for $7.75 per share, or about $275 million, inclusive of cash and debt. The deal ends the seven-year reprieve that kept the SGI banner flying after Rackable Systems purchased the bankrupt Silicon Graphics Inc. for $25 million in 2009 and assumed the SGI brand. Bringing SGI into its fold bolsters HPE's high-performance computing and data analytics capabilities and expands its position... Read more…

By Tiffany Trader

ARM Unveils Scalable Vector Extension for HPC at Hot Chips

August 22, 2016

ARM and Fujitsu today announced a scalable vector extension (SVE) to the ARMv8-A architecture intended to enhance ARM capabilities in HPC workloads. Fujitsu is the lead silicon partner in the effort (so far) and will use ARM with SVE technology in its post K computer, Japan’s next flagship supercomputer planned for the 2020 timeframe. This is an important incremental step for ARM, which seeks to push more aggressively into mainstream and HPC server markets. Read more…

By John Russell

IBM Debuts Power8 Chip with NVLink and Three New Systems

September 8, 2016

Not long after revealing more details about its next-gen Power9 chip due in 2017, IBM today rolled out three new Power8-based Linux servers and a new version of its Power8 chip featuring Nvidia’s NVLink interconnect. Read more…

By John Russell

Leading Solution Providers

Vectors: How the Old Became New Again in Supercomputing

September 26, 2016

Vector instructions, once a powerful performance innovation of supercomputing in the 1970s and 1980s became an obsolete technology in the 1990s. But like the mythical phoenix bird, vector instructions have arisen from the ashes. Here is the history of a technology that went from new to old then back to new. Read more…

By Lynd Stringer

US, China Vie for Supercomputing Supremacy

November 14, 2016

The 48th edition of the TOP500 list is fresh off the presses and while there is no new number one system, as previously teased by China, there are a number of notable entrants from the US and around the world and significant trends to report on. Read more…

By Tiffany Trader

Intel Launches Silicon Photonics Chip, Previews Next-Gen Phi for AI

August 18, 2016

At the Intel Developer Forum, held in San Francisco this week, Intel Senior Vice President and General Manager Diane Bryant announced the launch of Intel's Silicon Photonics product line and teased a brand-new Phi product, codenamed "Knights Mill," aimed at machine learning workloads. Read more…

By Tiffany Trader

CPU Benchmarking: Haswell Versus POWER8

June 2, 2015

With OpenPOWER activity ramping up and IBM’s prominent role in the upcoming DOE machines Summit and Sierra, it’s a good time to look at how the IBM POWER CPU stacks up against the x86 Xeon Haswell CPU from Intel. Read more…

By Tiffany Trader

Beyond von Neumann, Neuromorphic Computing Steadily Advances

March 21, 2016

Neuromorphic computing – brain inspired computing – has long been a tantalizing goal. The human brain does with around 20 watts what supercomputers do with megawatts. And power consumption isn’t the only difference. Fundamentally, brains ‘think differently’ than the von Neumann architecture-based computers. While neuromorphic computing progress has been intriguing, it has still not proven very practical. Read more…

By John Russell

Dell EMC Engineers Strategy to Democratize HPC

September 29, 2016

The freshly minted Dell EMC division of Dell Technologies is on a mission to take HPC mainstream with a strategy that hinges on engineered solutions, beginning with a focus on three industry verticals: manufacturing, research and life sciences. "Unlike traditional HPC where everybody bought parts, assembled parts and ran the workloads and did iterative engineering, we want folks to focus on time to innovation and let us worry about the infrastructure," said Jim Ganthier, senior vice president, validated solutions organization at Dell EMC Converged Platforms Solution Division. Read more…

By Tiffany Trader

Container App ‘Singularity’ Eases Scientific Computing

October 20, 2016

HPC container platform Singularity is just six months out from its 1.0 release but already is making inroads across the HPC research landscape. It's in use at Lawrence Berkeley National Laboratory (LBNL), where Singularity founder Gregory Kurtzer has worked in the High Performance Computing Services (HPCS) group for 16 years. Read more…

By Tiffany Trader

Micron, Intel Prepare to Launch 3D XPoint Memory

August 16, 2016

Micron Technology used last week’s Flash Memory Summit to roll out its new line of 3D XPoint memory technology jointly developed with Intel while demonstrating the technology in solid-state drives. Micron claimed its Quantx line delivers PCI Express (PCIe) SSD performance with read latencies at less than 10 microseconds and writes at less than 20 microseconds. Read more…

By George Leopold

  • arrow
  • Click Here for More Headlines
  • arrow
Share This